Oct 30 20:57:29.109898 ip-10-0-137-72 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 20:57:29.109910 ip-10-0-137-72 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 20:57:29.109919 ip-10-0-137-72 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 20:57:29.110212 ip-10-0-137-72 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 20:57:39.134210 ip-10-0-137-72 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 20:57:39.134229 ip-10-0-137-72 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 802a7bb79f094f6f9a0f61107a36ca64 --
Oct 30 20:59:43.083787 ip-10-0-137-72 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:59:43.526472 ip-10-0-137-72 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:59:43.526472 ip-10-0-137-72 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:59:43.526472 ip-10-0-137-72 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:59:43.526472 ip-10-0-137-72 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:59:43.532761 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532604    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.532761 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532757    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.532761 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532761    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.532761 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532764    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.532761 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532768    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532771    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532775    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532777    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532780    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532783    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532786    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532788    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532791    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532794    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532796    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532799    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532802    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532804    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532807    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532810    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532812    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532815    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532817    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.532972 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532820    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532822    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532825    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532827    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532830    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532832    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532835    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532837    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532840    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532847    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532849    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532852    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532855    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532857    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532861    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532864    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532867    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532870    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532873    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532875    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.533412 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532878    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532881    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532883    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532886    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532889    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532892    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532894    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532897    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532900    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532903    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532906    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532908    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532911    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532915    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532920    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532923    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532930    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532933    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.533906 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532935    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532939    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532942    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532944    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532947    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532949    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532952    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532956    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532959    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532962    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532966    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532970    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532973    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532976    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532979    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532982    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532985    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532988    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532991    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.534340 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532994    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.532996    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533000    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533003    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533006    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533009    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533011    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533014    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533017    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533020    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533022    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533025    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533028    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533031    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533033    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533036    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533039    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533041    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533044    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533048    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.534809 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533051    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533055    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533058    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533060    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533063    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533914    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533923    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533927    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533930    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533933    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533936    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533938    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533941    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533945    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533947    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533951    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533954    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533957    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533961    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.535272 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533966    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533969    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533972    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533976    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533979    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533983    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533988    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533993    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.533997    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534001    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534005    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534011    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534017    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534020    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534023    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534026    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534029    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534032    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534034    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.535752 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534037    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534040    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534042    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534045    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534048    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534050    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534053    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534056    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534059    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534061    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534064    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534067    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534070    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534073    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534075    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534078    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534080    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534083    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534086    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534088    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.536215 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534091    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534094    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534097    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534099    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534102    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534104    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534107    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534110    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534113    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534115    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534118    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534120    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534123    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534125    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534128    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534130    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534133    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534136    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534138    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534141    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.536712 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534143    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534146    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534149    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534152    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534154    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534157    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534159    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534162    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534164    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534167    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534170    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534173    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534175    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534178    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534181    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534183    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534186    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534188    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534191    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534194    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.537190 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534196    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534199    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534201    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534204    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534207    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534209    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534212    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534215    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534217    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534220    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534222    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534225    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534227    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534737    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534742    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534752    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534755    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534758    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534761    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.537678 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534763    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534766    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534769    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534772    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534774    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534777    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534779    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534783    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534785    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534788    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534790    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534793    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534797    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534800    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534803    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534805    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534808    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534811    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534813    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.538150 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534816    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534819    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534822    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534825    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534827    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534830    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534832    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534835    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534837    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534840    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534848    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534851    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534854    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534857    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534860    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534862    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534865    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534867    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534870    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534873    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.538615 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534875    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534878    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534881    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534883    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534886    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534889    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534891    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534894    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534905    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534908    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534911    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534913    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534916    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534919    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534921    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534924    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534926    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534929    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534931    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534934    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.539534 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534937    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534939    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534942    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534945    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534948    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534950    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534953    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534956    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534958    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534961    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534964    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534966    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534969    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534971    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534974    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534976    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534979    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534981    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534984    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534987    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.540060 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534989    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534992    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534995    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.534997    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535000    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535002    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535005    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535007    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535010    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535012    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535015    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535017    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535020    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535022    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535025    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535029    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535033    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535036    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535039    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.540555 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535045    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.535049    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536244    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536255    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536266    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536271    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536276    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536279    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536283    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536288    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536292    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536295    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536298    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536302    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536305    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536308    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536311    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536314    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536317    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536320    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536326    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536330    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536333    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536336    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:59:43.541043 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536339    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536344    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536347    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536350    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536354    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536357    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536362    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536366    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536369    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536372    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536377    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536380    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536383    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536386    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536389    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536392    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536397    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536401    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536404    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536407    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536410    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536414    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536417    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536421    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536424    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:59:43.541660 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536427    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536430    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536433    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536436    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536439    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536442    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536445    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536449    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536452    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536455    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536459    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536462    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536465    2581 flags.go:64] FLAG: --help="false"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536469    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536472    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536476    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536479    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536482    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536486    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536501    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536504    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536507    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536511    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:59:43.542277 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536514    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536517    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536520    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536523    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536526    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536528    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536531    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536534    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536537    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536540    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536543    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536549    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536552    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536554    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536558    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536560    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536564    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536567    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536570    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536574    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536577    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536582    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536585    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536588    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536591    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:59:43.542841 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536594    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536598    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536600    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536603    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536611    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536614    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536617    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536620    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536623    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536626    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536629    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536632    2581 flags.go:64] FLAG: --port="10250"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536635    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536638    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea9ad03d3f4cd32a"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536641    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536644    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536647    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536650    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536654    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536656    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536659    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536662    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536666    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536669    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536672    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536675    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:59:43.543418 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536679    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536682    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536685    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536688    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536691    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536696    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536700    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536703    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536707    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536710    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536713    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536716    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536719    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536722    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536725    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536728    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536734    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536737    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536739    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536743    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536746    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536749    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536752    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536755    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536758    2581 flags.go:64] FLAG: --v="2"
Oct 30 20:59:43.544088 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536763    2581 flags.go:64] FLAG: --version="false"
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536768    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536772    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.536776    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536911    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536914    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536918    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536921    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536924    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536929    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536932    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536936    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536938    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536947    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536950    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536953    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536955    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536958    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536961    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.544706 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536963    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536966    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536968    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536971    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536974    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536977    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536980    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536982    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536985    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536987    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536990    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536993    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.536996    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537001    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537003    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537006    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537009    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537011    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537014    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537016    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.545206 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537019    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537021    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537024    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537027    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537034    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537037    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537039    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537048    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537051    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537053    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537056    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537059    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537062    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537065    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537069    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537072    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537075    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537078    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537081    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.545704 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537083    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537086    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537088    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537091    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537094    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537096    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537100    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537103    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537106    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537108    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537111    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537114    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537116    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537119    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537121    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537124    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537126    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537130    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537133    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.546155 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537136    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537140    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537153    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537156    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537160    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537162    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537165    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537167    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537170    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537173    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537175    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537178    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537180    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537183    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537186    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537188    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537191    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537194    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.546634 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537197    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537200    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537203    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537206    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537209    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537211    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537214    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537217    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537219    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537222    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537225    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537228    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537232    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537234    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.537261    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537307    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.547061 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537312    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537317    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537320    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537323    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537326    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537329    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537331    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537334    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537337    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537339    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537342    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537344    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537347    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537349    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537352    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537355    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537358    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537360    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537364    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.547435 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537367    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537370    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537372    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537376    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537378    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537381    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537384    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537387    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537389    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537393    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537396    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537398    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537401    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537404    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537406    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537409    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537412    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537415    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537417    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537420    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.547944 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537422    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537425    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537428    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537430    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537432    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537449    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537453    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537456    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537459    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537462    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537466    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537470    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537473    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537475    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537478    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537480    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537483    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537486    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537500    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537503    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.548520 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537505    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537509    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537512    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537514    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537517    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537520    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537523    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537526    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537529    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537532    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537534    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537537    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537540    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537542    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537545    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537548    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537550    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537554    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537558    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.549000 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537562    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537565    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537567    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537570    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537574    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537576    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537579    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537582    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537584    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537587    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537589    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537592    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537595    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537597    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537601    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537604    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537606    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537609    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537612    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537614    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.549468 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537617    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537619    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537622    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537626    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537630    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537633    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.537638    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.544456    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.544473    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.544509    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.544515    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544576    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544583    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544586    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544589    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544592    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544595    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544598    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.549969 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544601    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544604    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544607    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544610    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544613    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544615    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544618    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544621    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544624    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544626    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544629    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544632    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544635    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544637    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544640    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544643    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544646    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544649    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544651    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544654    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.550446 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544657    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544660    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544668    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544671    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544674    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544677    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544679    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544682    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544685    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544687    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544690    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544693    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544695    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544698    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544701    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544704    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544706    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544709    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544712    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544715    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.550947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544718    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544721    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544723    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544726    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544729    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544731    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544735    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544737    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544740    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544743    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544745    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544748    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544751    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544754    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544758    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544769    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544772    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544775    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544778    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.551416 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544780    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544783    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544787    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544791    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544794    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544797    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544800    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544802    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544805    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544807    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544810    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544813    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544815    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544818    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544821    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544824    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544826    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544828    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544831    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.552009 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544835    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544838    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544840    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544843    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544846    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544848    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544851    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544854    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544857    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544859    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544868    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544871    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544873    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544876    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544879    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544882    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544885    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544887    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544890    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544892    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.552472 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.544895    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.544922    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545110    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545114    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545117    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545120    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545122    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545125    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545128    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545130    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545133    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545136    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545139    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545142    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545145    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545147    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.552966 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545150    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545153    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545156    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545158    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545161    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545164    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545169    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545178    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545182    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545186    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545189    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545191    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545194    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545197    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545200    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545202    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545205    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545208    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545211    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545214    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.553415 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545216    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545219    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545222    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545225    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545227    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545230    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545232    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545235    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545238    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545241    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545244    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545246    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545249    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545252    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545255    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545257    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545260    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545263    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545266    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.553962 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545269    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545277    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545280    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545283    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545286    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545288    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545291    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545294    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545296    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545299    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545301    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545304    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545307    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545309    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545312    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545314    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545317    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545320    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545322    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.554791 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545325    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545328    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545330    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545333    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545343    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545345    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545348    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545350    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545353    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545356    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545358    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545361    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545363    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545366    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545369    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545377    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545379    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545382    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545384    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.555288 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545387    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545391    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545394    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545397    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545400    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545403    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545406    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545409    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545411    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545414    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545416    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545419    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545422    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545424    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545427    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.555783 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.545449    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545499    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545505    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545507    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545510    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545513    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545516    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545519    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545521    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545525    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545527    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545530    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545533    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545535    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545544    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545547    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545549    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545552    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545554    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:59:43.556186 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545557    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545559    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545562    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545564    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545567    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545570    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545573    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545577    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545580    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545583    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545586    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545588    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545591    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545594    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545596    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545599    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545601    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545604    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545606    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:59:43.556716 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545609    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545612    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545614    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545617    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545619    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545622    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545624    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545627    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545629    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545638    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545640    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545643    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545645    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545648    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545650    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545653    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545656    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545658    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545661    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545663    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:59:43.557295 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545666    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545670    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545673    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545676    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545679    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545682    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545685    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545688    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545690    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545693    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545696    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545699    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545701    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545704    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545706    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545709    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545712    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545715    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545718    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545721    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:59:43.557785 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545723    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545726    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545734    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545737    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545739    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545742    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545744    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545747    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545749    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545752    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545755    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545757    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545760    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545762    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545765    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545767    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545770    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545773    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545776    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:59:43.558458 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545778    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545781    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545784    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545786    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545789    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545791    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545795    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545797    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545800    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:43.545803    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.545870    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.547930    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.549135    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.549230    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.549884    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:59:43.558929 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.552768    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:59:43.559279 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.558951    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:59:43.569632 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.569612    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:59:43.572919 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.572903    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 20:59:43.573758 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.573729    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:59:43.576581 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.576562    2581 fs.go:135] Filesystem UUIDs: map[2f2bb360-add7-46f6-aa12-10d78fc81670:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb38f94a-a4c0-4cd8-b3c5-7865e325bdd7:/dev/nvme0n1p4]
Oct 30 20:59:43.576644 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.576580    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:59:43.582128 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.582014    2581 manager.go:217] Machine: {Timestamp:2026-10-30 20:59:43.58067548 +0000 UTC m=+0.370829009 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096551 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6efa917121a345c1956545e75325 SystemUUID:ec2c6efa-9171-21a3-45c1-956545e75325 BootID:802a7bb7-9f09-4f6f-9a0f-61107a36ca64 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:25:18:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:25:18:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:ca:8b:0e:42:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:59:43.582203 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.582131    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:59:43.582232 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.582209    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:59:43.583458 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.583424    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:59:43.583672 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.583462    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:59:43.584282 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.584269    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:59:43.584319 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.584286    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:59:43.584346 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.584327    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:59:43.584376 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.584346    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:59:43.585229 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.585217    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:59:43.589263 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.589250    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:59:43.589462 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.589451    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:59:43.590484 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590472    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:59:43.590545 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590502    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:59:43.590545 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590516    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:59:43.590545 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590527    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:59:43.590545 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590538    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:59:43.590987 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.590971    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:59:43.593815 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.593784    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:59:43.598569 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.598555    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:59:43.599538 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599524    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:59:43.599583 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599563    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:59:43.599771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599760    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:59:43.599802 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599781    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:59:43.599802 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599790    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:59:43.599802 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599798    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:59:43.599873 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599806    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:59:43.599873 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.599814    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:59:43.600441 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600431    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:59:43.600469 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600445    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:59:43.600469 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600454    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:59:43.600469 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600463    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:59:43.600589 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600481    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:59:43.600589 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.600510    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:59:43.601092 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.601082    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:59:43.601123 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.601096    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:59:43.605466 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.605452    2581 server.go:1263] "Started kubelet"
Oct 30 20:59:43.605575 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.605524    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:59:43.605729 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.605673    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:59:43.605797 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.605765    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:59:43.606326 ip-10-0-137-72 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:59:43.606901 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.606884    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:59:43.611617 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.611594    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:59:43.611944 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.611928    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:59:43.617035 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.617014    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:59:43.617547 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.617527    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:59:43.617663 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.617637    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:43.618363 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618323    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:59:43.618363 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618361    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:59:43.618529 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618451    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:59:43.618587 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618532    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:59:43.618587 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618540    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:59:43.618942 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618924    2581 factory.go:55] Registering systemd factory
Oct 30 20:59:43.618942 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.618943    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:59:43.619169 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.619155    2581 factory.go:153] Registering CRI-O factory
Oct 30 20:59:43.619169 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.619170    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:59:43.619287 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.619216    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:59:43.619287 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.619238    2581 factory.go:103] Registering Raw factory
Oct 30 20:59:43.619287 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.619253    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:59:43.620780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.620763    2581 manager.go:319] Starting recovery of all containers
Oct 30 20:59:43.630994 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.630873    2581 manager.go:324] Recovery completed
Oct 30 20:59:43.635005 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.634990    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.636820 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.636797    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.636889 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.636833    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.636889 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.636850    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.637375 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.637361    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:59:43.637375 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.637367    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.637375 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.637371    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:59:43.637568 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.637386    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:59:43.638284 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.638270    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.638352 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.638300    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.638352 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.638317    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.639738 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.639725    2581 policy_none.go:50] "Start"
Oct 30 20:59:43.639788 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.639741    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:59:43.639788 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.639755    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:59:43.641791 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.641778    2581 policy_none.go:44] "Start"
Oct 30 20:59:43.644694 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.644669    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:59:43.644974 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.644953    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-72.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:59:43.648018 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.644612    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-72.ec2.internal.18e36a53871fb492  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-72.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-72.ec2.internal,},FirstTimestamp:2026-10-30 20:59:43.605421202 +0000 UTC m=+0.395574735,LastTimestamp:2026-10-30 20:59:43.605421202 +0000 UTC m=+0.395574735,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-72.ec2.internal,}"
Oct 30 20:59:43.650143 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.650127    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:59:43.650200 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.650168    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2dglp"
Oct 30 20:59:43.657976 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.657947    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:59:43.657976 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.657967    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:59:43.658116 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.657994    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 20:59:43.677445 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.677398    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:59:43.681577 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.681563    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:59:43.681642 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.681595    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:59:43.681642 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.681604    2581 server.go:87] "Starting device plugin registration server"
Oct 30 20:59:43.681853 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.681840    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:59:43.681893 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.681854    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:59:43.681945 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.681932    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:59:43.682065 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.682003    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:59:43.682065 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.682014    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:59:43.682921 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.682846    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:59:43.682921 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.682897    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:43.763827 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.763800    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:59:43.765817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.765041    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:59:43.765817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.765060    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:59:43.765817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.765080    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:59:43.765817 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.765114    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:59:43.780160 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.780115    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:59:43.782212 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.782199    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.783009 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.782991    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.783111 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.783021    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.783111 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.783032    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.783111 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.783055    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.801988 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.801973    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.802055 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.801995    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-72.ec2.internal\": node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:43.837500 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.837457    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:43.866034 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.866011    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"]
Oct 30 20:59:43.866129 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.866064    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.866910 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.866894    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.866996 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.866921    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.866996 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.866931    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.869040 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869028    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.869182 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.869222 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869199    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.869785 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869769    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.869895 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869784    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.869895 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869795    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.869895 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869808    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.869895 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869821    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.869895 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.869865    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.871867 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.871850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.871970 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.871880    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:43.872571 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.872557    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:43.872636 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.872586    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:43.872636 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:43.872600    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:43.898971 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.898955    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-72.ec2.internal\" not found" node="ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.901753 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.901737    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-72.ec2.internal\" not found" node="ip-10-0-137-72.ec2.internal"
Oct 30 20:59:43.937935 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:43.937897    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.020151 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.020080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.020252 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.020169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.020252 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.020193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.038188 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.038137    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.120830 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.120927 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.120927 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.121019 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120925    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.121019 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120928    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.121019 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.120928    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.138821 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.138799    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.205527 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.205473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.205655 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.205486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.239576 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.239543    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.340874 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.340787    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.441100 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.441061    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.541684 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.541649    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.548264 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.548239    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:59:44.548397 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.548376    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:59:44.642070 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.642007    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.655218 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.655183    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:54:44 +0000 UTC" deadline="2028-07-23 04:19:37.19146298 +0000 UTC"
Oct 30 20:59:44.655218 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.655213    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15151h19m52.536253096s"
Oct 30 20:59:44.742600 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.742561    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.792611 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:44.792563    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe11cf10dd8daf22a5ab8288141ce668.slice/crio-22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf WatchSource:0}: Error finding container 22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf: Status 404 returned error can't find the container with id 22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf
Oct 30 20:59:44.798231 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:44.798205    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4d5a6e1cac786a378b125c5389b555.slice/crio-ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e WatchSource:0}: Error finding container ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e: Status 404 returned error can't find the container with id ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e
Oct 30 20:59:44.799837 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.799817    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:59:44.814097 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.814076    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:59:44.821526 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.821508    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:59:44.822773 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.822755    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 20:59:44.822844 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.822789    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 20:59:44.822844 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.822799    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 20:59:44.843215 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:44.843188    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 20:59:44.902032 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.901966    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:59:44.919317 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.919276    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.945840 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.945817    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:59:44.948162 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.948146    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 20:59:44.971249 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:44.971225    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:59:45.110943 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.109688    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:59:45.591236 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.591196    2581 apiserver.go:52] "Watching apiserver"
Oct 30 20:59:45.617724 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.617693    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:59:45.640118 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.640083    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:59:45.640410 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.640391    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zkpxs","openshift-cluster-node-tuning-operator/tuned-k5ct6","openshift-image-registry/node-ca-tfthv","openshift-multus/multus-tzlnj","openshift-network-diagnostics/network-check-target-nnm9g","kube-system/global-pull-secret-syncer-n5gz6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv","openshift-multus/network-metrics-daemon-7bqg5","openshift-network-operator/iptables-alerter-qtlzt","openshift-dns/node-resolver-rh2wd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-xgsjb","kube-system/konnectivity-agent-9fkqb","kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"]
Oct 30 20:59:45.642963 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.642934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.645680 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.645652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.647923 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.647901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.650255 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.650235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.653599 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.653578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:45.653699 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.653676    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:45.656279 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.656250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.656384 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.656316    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:45.658984 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.658615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.661412 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.661058    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.661412 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.661197    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.661412 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.661308    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4x2zw\""
Oct 30 20:59:45.661412 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.661369    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:59:45.663382 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.662338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.663382 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.662581    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:59:45.663382 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.662715    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:59:45.663382 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.662972    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tzc2p\""
Oct 30 20:59:45.664431 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.664413    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.665314 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.664838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.668364 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.667781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.670094 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.669975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.670094 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.670048    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:45.670620 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.670581    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:59:45.672611 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.672590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.680318 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.680296    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:59:45.680419 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.680374    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f9pc9"
Oct 30 20:59:45.719544 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.719508    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:59:45.735067 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zb2\" (UniqueName: \"kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.735195 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bz8\" (UniqueName: \"kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.735417 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.735789 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.735789 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.735854 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.735854 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.735924 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.735924 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.735924 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736029 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736029 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.735975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.736029 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97x6l\" (UniqueName: \"kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.736136 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736136 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736136 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.736238 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736238 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736309 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736309 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.736309 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.736309 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736433 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736433 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckkd4\" (UniqueName: \"kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.736433 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2sb\" (UniqueName: \"kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.736531 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736531 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.736531 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.736663 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.736663 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736663 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736663 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.736784 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm44\" (UniqueName: \"kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.736784 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736784 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736784 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736784 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bjx\" (UniqueName: \"kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.736935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737115 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737115 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737115 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.736998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.737115 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.737229 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737229 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737229 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737322 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737322 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737322 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.737322 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.737322 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggpw\" (UniqueName: \"kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vh7w\" (UniqueName: \"kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.737482 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.737880 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.737880 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.737880 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.737521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.770737 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.770676    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerStarted","Data":"22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf"}
Oct 30 20:59:45.770932 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.770858    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.770932 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.770871    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:59:45.770932 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.770895    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.770932 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.770898    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.771218 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771203    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.771274 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771262    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8v8x\""
Oct 30 20:59:45.771343 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771331    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:59:45.771463 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771438    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7r6m\""
Oct 30 20:59:45.771533 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771489    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:59:45.771631 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771616    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.771825 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.771812    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.772053 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772016    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:59:45.772053 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772042    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772086    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772098    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772124    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772158    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772166    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:59:45.772215 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772204    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.772504 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772229    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hvtl2\""
Oct 30 20:59:45.772504 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772322    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.772504 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772386    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:59:45.772504 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772398    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.772988 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.772958    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal" event={"ID":"ca4d5a6e1cac786a378b125c5389b555","Type":"ContainerStarted","Data":"ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e"}
Oct 30 20:59:45.785668 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.785643    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-79n9v\""
Oct 30 20:59:45.785668 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.785649    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gszwc\""
Oct 30 20:59:45.786234 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.786203    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnqjk\""
Oct 30 20:59:45.786339 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.786281    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8vzq\""
Oct 30 20:59:45.786339 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.786288    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:59:45.837990 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.837954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.837997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm44\" (UniqueName: \"kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838120    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838200    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bjx\" (UniqueName: \"kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838251    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838671    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.838817 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838879    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838928    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839201    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839266    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.838566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.839616    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839667    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839672    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.839695    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:46.33967234 +0000 UTC m=+3.129825878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839727    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839727    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.839771 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839877    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839894    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggpw\" (UniqueName: \"kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.839987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh7w\" (UniqueName: \"kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840134    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840181    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.840510 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840232    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840321    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840329    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zb2\" (UniqueName: \"kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840410    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840475    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840511    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.841354 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840518    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840556    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bz8\" (UniqueName: \"kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.840675    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.840740    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:46.340720806 +0000 UTC m=+3.130874341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840880    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840930    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.840959    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841154    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841229    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.842179 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841292    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841301    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97x6l\" (UniqueName: \"kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843071 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841706    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841852    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841867    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842200    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842350    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.841757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842390    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842458    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.843953 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842524    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkd4\" (UniqueName: \"kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2sb\" (UniqueName: \"kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842703    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842712    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842667    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842746    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.844876 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842915    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842933    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842945    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.842994    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.843424    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.843425    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.844126    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.845780 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.844412    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.849651 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.847025    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.849651 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.849528    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.861402 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.861374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm44\" (UniqueName: \"kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.862950 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.862764    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:59:45.862950 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.862821    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:59:45.862950 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.862835    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:45.862950 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:45.862890    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:46.362872203 +0000 UTC m=+3.153025735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:45.863359 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.863331    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggpw\" (UniqueName: \"kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:45.863628 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.863604    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bjx\" (UniqueName: \"kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:45.865191 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.865165    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2sb\" (UniqueName: \"kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.865960 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.865934    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh7w\" (UniqueName: \"kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:45.866346 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.866329    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zb2\" (UniqueName: \"kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.866346 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.866336    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97x6l\" (UniqueName: \"kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:45.867574 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.867298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bz8\" (UniqueName: \"kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:45.867574 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.867501    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkd4\" (UniqueName: \"kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:45.962867 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.962833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 20:59:45.966036 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.966010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:45.977313 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:45.977282    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfbaf77_0475_4cc0_ad2c_3e1cfe49e593.slice/crio-d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c WatchSource:0}: Error finding container d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c: Status 404 returned error can't find the container with id d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c
Oct 30 20:59:45.978222 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:45.978195    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7871fe29_fbbf_4733_878f_680fb56cee31.slice/crio-b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e WatchSource:0}: Error finding container b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e: Status 404 returned error can't find the container with id b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e
Oct 30 20:59:45.983362 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.983324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfthv"
Oct 30 20:59:45.991856 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:45.991832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tzlnj"
Oct 30 20:59:45.994947 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:45.994848    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56952e8_b525_401d_8128_e4c8ac0d6469.slice/crio-20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81 WatchSource:0}: Error finding container 20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81: Status 404 returned error can't find the container with id 20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81
Oct 30 20:59:46.003101 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.003074    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60e19dc_42fc_4ad2_a73b_bb6f3bfc651f.slice/crio-da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417 WatchSource:0}: Error finding container da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417: Status 404 returned error can't find the container with id da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417
Oct 30 20:59:46.003209 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.003117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 20:59:46.014274 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.014242    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76c52e5_2903_48df_92ea_bd07b934c773.slice/crio-e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a WatchSource:0}: Error finding container e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a: Status 404 returned error can't find the container with id e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a
Oct 30 20:59:46.025210 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.025180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 20:59:46.028403 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.028376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rh2wd"
Oct 30 20:59:46.032779 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.032756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 20:59:46.036868 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.036846    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ac9b7b_1e1f_4966_8522_d92af9ee73b3.slice/crio-5c7e3a3300354c3eb0a21969ba40ea8f5700e42a23a5a4b65047381b5d79d26f WatchSource:0}: Error finding container 5c7e3a3300354c3eb0a21969ba40ea8f5700e42a23a5a4b65047381b5d79d26f: Status 404 returned error can't find the container with id 5c7e3a3300354c3eb0a21969ba40ea8f5700e42a23a5a4b65047381b5d79d26f
Oct 30 20:59:46.041090 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.041066    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b36017a_ec01_4979_a358_7198261684d7.slice/crio-5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd WatchSource:0}: Error finding container 5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd: Status 404 returned error can't find the container with id 5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd
Oct 30 20:59:46.043715 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.043691    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e300873_ed21_450b_9e6b_810ee569b783.slice/crio-31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb WatchSource:0}: Error finding container 31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb: Status 404 returned error can't find the container with id 31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb
Oct 30 20:59:46.050616 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.050592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 20:59:46.061729 ip-10-0-137-72 kubenswrapper[2581]: W1030 20:59:46.061702    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebd2d0e_ac9b_4ffb_9da5_bb5cc6b391db.slice/crio-baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4 WatchSource:0}: Error finding container baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4: Status 404 returned error can't find the container with id baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4
Oct 30 20:59:46.347018 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.346971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:46.347197 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.347032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:46.347197 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.347151    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:46.347197 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.347153    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:46.347379 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.347220    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:47.347202201 +0000 UTC m=+4.137355732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:46.347379 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.347241    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:47.347231028 +0000 UTC m=+4.137384548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:46.448116 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.447885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:46.448325 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.448306    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:59:46.448402 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.448334    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:59:46.448402 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.448347    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:46.448546 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.448412    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:47.448391679 +0000 UTC m=+4.238545199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:46.690078 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.689754    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:46 +0000 UTC" deadline="2028-04-19 12:31:16.095575777 +0000 UTC"
Oct 30 20:59:46.690078 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.689820    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12879h31m29.405760743s"
Oct 30 20:59:46.766575 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.765932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:46.766575 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:46.766091    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:46.799173 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.798968    2581 generic.go:356] "Generic (PLEG): container finished" podID="be11cf10dd8daf22a5ab8288141ce668" containerID="52d3bbc0c5bcb0c50f2efc6d172b86ec152c3b92777ff246161dc37825313405" exitCode=0
Oct 30 20:59:46.799173 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.799112    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerDied","Data":"52d3bbc0c5bcb0c50f2efc6d172b86ec152c3b92777ff246161dc37825313405"}
Oct 30 20:59:46.802303 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.802242    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerStarted","Data":"31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb"}
Oct 30 20:59:46.807206 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.807117    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rh2wd" event={"ID":"2b36017a-ec01-4979-a358-7198261684d7","Type":"ContainerStarted","Data":"5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd"}
Oct 30 20:59:46.810180 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.810122    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tzlnj" event={"ID":"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f","Type":"ContainerStarted","Data":"da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417"}
Oct 30 20:59:46.815749 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.815700    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fkqb" event={"ID":"7871fe29-fbbf-4733-878f-680fb56cee31","Type":"ContainerStarted","Data":"b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e"}
Oct 30 20:59:46.820712 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.820549    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4"}
Oct 30 20:59:46.822381 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.822316    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5ct6" event={"ID":"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3","Type":"ContainerStarted","Data":"5c7e3a3300354c3eb0a21969ba40ea8f5700e42a23a5a4b65047381b5d79d26f"}
Oct 30 20:59:46.827092 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.827013    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qtlzt" event={"ID":"e76c52e5-2903-48df-92ea-bd07b934c773","Type":"ContainerStarted","Data":"e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a"}
Oct 30 20:59:46.831291 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.831227    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfthv" event={"ID":"c56952e8-b525-401d-8128-e4c8ac0d6469","Type":"ContainerStarted","Data":"20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81"}
Oct 30 20:59:46.836055 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.835950    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv" event={"ID":"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593","Type":"ContainerStarted","Data":"d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c"}
Oct 30 20:59:46.850366 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:46.849715    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal" event={"ID":"ca4d5a6e1cac786a378b125c5389b555","Type":"ContainerStarted","Data":"2e65360950122a7e50dc09737efdfb5c196ca34b1ac9e5d27c627208e52f56e9"}
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.355186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.355297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.355420    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.355486    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:49.355467246 +0000 UTC m=+6.145620767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.355916    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:47.356058 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.355969    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:49.355953787 +0000 UTC m=+6.146107306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:47.457138 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.456397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:47.457138 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.456589    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:59:47.457138 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.456613    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:59:47.457138 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.456628    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:47.457138 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.456721    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:49.456669677 +0000 UTC m=+6.246823207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:47.690745 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.690650    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:46 +0000 UTC" deadline="2028-06-22 08:48:22.974961503 +0000 UTC"
Oct 30 20:59:47.690745 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.690702    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14411h48m35.284264703s"
Oct 30 20:59:47.773171 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.768747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:47.773171 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.768881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:47.773171 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.769304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:47.773171 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:47.769402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:47.865760 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.865723    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerStarted","Data":"981bbf3fa74d9811e2a13115e142c9531239cf3f29349e74da13bd27be96f72a"}
Oct 30 20:59:47.912694 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:47.912630    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal" podStartSLOduration=2.9126091990000003 podStartE2EDuration="2.912609199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:46.892315517 +0000 UTC m=+3.682469060" watchObservedRunningTime="2026-10-30 20:59:47.912609199 +0000 UTC m=+4.702762744"
Oct 30 20:59:48.767093 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:48.766277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:48.767093 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:48.766411    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:49.372914 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.372828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:49.372914 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.372885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:49.373141 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.373074    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:49.373141 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.373139    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:53.373118855 +0000 UTC m=+10.163272377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:49.373695 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.373669    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:49.373810 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.373733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:53.373716295 +0000 UTC m=+10.163869814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:49.474164 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.473536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:49.474164 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.473676    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:59:49.474164 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.473693    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:59:49.474164 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.473704    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:49.474164 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.473749    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:53.473736343 +0000 UTC m=+10.263889860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:49.766811 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.766778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:49.766979 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.766928    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:49.767504 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.767469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:49.767964 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:49.767604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:49.876636 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.876006    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="8e6e62b073e941864ed45b85baaac6a2a3838c2aa7f4002225edaa1d1206aebe" exitCode=0
Oct 30 20:59:49.876636 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.876073    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"8e6e62b073e941864ed45b85baaac6a2a3838c2aa7f4002225edaa1d1206aebe"}
Oct 30 20:59:49.950633 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:49.950107    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" podStartSLOduration=4.950090372 podStartE2EDuration="4.950090372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:47.913507752 +0000 UTC m=+4.703661287" watchObservedRunningTime="2026-10-30 20:59:49.950090372 +0000 UTC m=+6.740243914"
Oct 30 20:59:50.765889 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:50.765834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:50.766068 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:50.766009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:50.885048 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:50.885007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fkqb" event={"ID":"7871fe29-fbbf-4733-878f-680fb56cee31","Type":"ContainerStarted","Data":"d7fe1da512fa1f0ce4b117091636a74f3b7b8b7a9fc57ee37c5090bd8797b7d4"}
Oct 30 20:59:51.769807 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.769775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:51.770015 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:51.769925    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:51.770319 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.770297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:51.770423 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:51.770397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:51.889138 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.889100    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:51.890093 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.889882    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 20:59:51.892024 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.891621    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfthv" event={"ID":"c56952e8-b525-401d-8128-e4c8ac0d6469","Type":"ContainerStarted","Data":"61a2c07e31b59746993f4a9d6b1c2b78b75f4d074c4e15a214f014eea90a4dbc"}
Oct 30 20:59:51.931209 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:51.931163    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9fkqb" podStartSLOduration=4.189192489 podStartE2EDuration="7.931144921s" totalImagesPullingTime="3.741952432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:50.927230099 +0000 UTC m=+7.717383639" watchObservedRunningTime="2026-10-30 20:59:51.931144921 +0000 UTC m=+8.721298461"
Oct 30 20:59:52.766905 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:52.766303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:52.766905 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:52.766459    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:52.901310 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:52.901279    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:59:52.901842 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:52.901812    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv" event={"ID":"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593","Type":"ContainerStarted","Data":"d58e1555e1ddfb8b779870f79ab15e89767d3f415f71f43d26a574b1578467dd"}
Oct 30 20:59:53.408635 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.408587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:53.408825 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.408653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:53.408825 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.408804    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:53.408968 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.408883    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:01.408860934 +0000 UTC m=+18.199014456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:59:53.409043 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.408973    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:53.409043 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.409017    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:01.409004618 +0000 UTC m=+18.199158137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:59:53.509612 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.509571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:53.509819 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.509796    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:59:53.509887 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.509830    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:59:53.509887 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.509845    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:53.509950 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.509910    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:01.509891222 +0000 UTC m=+18.300044738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:59:53.770378 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.769824    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:53.770378 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.769989    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:53.770378 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.770034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:53.770378 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:53.770153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:53.868222 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.868188    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:59:53.906163 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:53.906060    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv" event={"ID":"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593","Type":"ContainerStarted","Data":"60bf73386c4f8283c82dd769b5c1db5e6e2ea8fcba39702cd9ed9793bdc33e96"}
Oct 30 20:59:54.693836 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.693474    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:59:53.868221536Z","UUID":"3f8202c1-d784-48ac-8ea4-38caf03873ee","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:59:54.695133 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.695098    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:59:54.695133 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.695131    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:59:54.766086 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.766008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:54.766246 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:54.766143    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:54.915921 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.915891    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="8429dd771c43723257c0d19f5e6cfebf6536b721a585b60fb9fcfc781e380b5f" exitCode=0
Oct 30 20:59:54.916461 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.915985    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"8429dd771c43723257c0d19f5e6cfebf6536b721a585b60fb9fcfc781e380b5f"}
Oct 30 20:59:54.919608 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.919401    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rh2wd" event={"ID":"2b36017a-ec01-4979-a358-7198261684d7","Type":"ContainerStarted","Data":"e4dc75b8269ee38ed9ff555f99a6d4f05688df6624529272eec6971b2c72a405"}
Oct 30 20:59:54.922290 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.922264    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tzlnj" event={"ID":"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f","Type":"ContainerStarted","Data":"3a099741745922aed54b5f0b40be0683324d6310fb7a590e554fb25e0c71bd15"}
Oct 30 20:59:54.964147 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:54.964086    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tfthv" podStartSLOduration=5.924294756 podStartE2EDuration="10.964069445s" totalImagesPullingTime="5.039774689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:51.972607877 +0000 UTC m=+8.762761414" watchObservedRunningTime="2026-10-30 20:59:54.964069445 +0000 UTC m=+11.754222989"
Oct 30 20:59:55.008890 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.008844    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tzlnj" podStartSLOduration=2.80014402 podStartE2EDuration="11.008828339s" totalImagesPullingTime="8.208684319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:55.00805805 +0000 UTC m=+11.798211593" watchObservedRunningTime="2026-10-30 20:59:55.008828339 +0000 UTC m=+11.798981879"
Oct 30 20:59:55.050240 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.050151    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rh2wd" podStartSLOduration=2.7074014589999997 podStartE2EDuration="11.050137178s" totalImagesPullingTime="8.342735719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:55.049840982 +0000 UTC m=+11.839994522" watchObservedRunningTime="2026-10-30 20:59:55.050137178 +0000 UTC m=+11.840290718"
Oct 30 20:59:55.766360 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.766332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:55.766553 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:55.766528    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:55.766994 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.766961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:55.767109 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:55.767047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:55.926899 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.926424    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qtlzt" event={"ID":"e76c52e5-2903-48df-92ea-bd07b934c773","Type":"ContainerStarted","Data":"ef04c752bde440ff60c844867fd36598c5b0eff57184947f9a5fbba81ff0e496"}
Oct 30 20:59:55.929998 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.929967    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv" event={"ID":"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593","Type":"ContainerStarted","Data":"347500303d2a3dc14a04cec6026a7ac022833b8c15c5c0fb2f0d27ceddd18041"}
Oct 30 20:59:55.932935 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.932900    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="45bd75c49893f17cc3f4ef48b36047b89e43d914de653ef951ec30ea5b5f0f70" exitCode=0
Oct 30 20:59:55.933576 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.933549    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"45bd75c49893f17cc3f4ef48b36047b89e43d914de653ef951ec30ea5b5f0f70"}
Oct 30 20:59:55.966998 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:55.966950    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qtlzt" podStartSLOduration=3.7661730909999998 podStartE2EDuration="11.966936032s" totalImagesPullingTime="8.200762941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:55.966884695 +0000 UTC m=+12.757038232" watchObservedRunningTime="2026-10-30 20:59:55.966936032 +0000 UTC m=+12.757089568"
Oct 30 20:59:56.047061 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.046997    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv" podStartSLOduration=3.659656757 podStartE2EDuration="12.046974623s" totalImagesPullingTime="8.387317866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:56.04664092 +0000 UTC m=+12.836794518" watchObservedRunningTime="2026-10-30 20:59:56.046974623 +0000 UTC m=+12.837128161"
Oct 30 20:59:56.765948 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.765916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:56.766123 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:56.766089    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:56.936519 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.936450    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5ct6" event={"ID":"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3","Type":"ContainerStarted","Data":"2d2559f981a61292f6b50282038360d63ab38a2b265b79c55be66a025682d68f"}
Oct 30 20:59:56.939336 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.939303    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="6048b16741b376a01ff81d2d5614350893e6c775b6ea6ebb8dc7c27310064f8e" exitCode=0
Oct 30 20:59:56.939593 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.939539    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"6048b16741b376a01ff81d2d5614350893e6c775b6ea6ebb8dc7c27310064f8e"}
Oct 30 20:59:56.975575 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:56.975517    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k5ct6" podStartSLOduration=2.279396565 podStartE2EDuration="12.975484706s" totalImagesPullingTime="10.696088141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:56.975117701 +0000 UTC m=+13.765271237" watchObservedRunningTime="2026-10-30 20:59:56.975484706 +0000 UTC m=+13.765638243"
Oct 30 20:59:57.766089 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:57.765860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:57.766262 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:57.766107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:57.766262 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:57.766198    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:57.766572 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:57.766548    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:58.766236 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:58.766212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 20:59:58.766636 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:58.766328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 20:59:58.947806 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:58.947661    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="934e17bc9087d81c06c6c3481f474f8507ed5cca1e0ae4e12b4860489ac78f38" exitCode=0
Oct 30 20:59:58.947806 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:58.947715    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"934e17bc9087d81c06c6c3481f474f8507ed5cca1e0ae4e12b4860489ac78f38"}
Oct 30 20:59:59.766069 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:59.766027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 20:59:59.766231 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:59.766135    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 20:59:59.766507 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:59.766480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 20:59:59.766849 ip-10-0-137-72 kubenswrapper[2581]: E1030 20:59:59.766591    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 20:59:59.953957 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:59.953922    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="e783eee8eb6fead7ab6db0e1267a635802dfa79afe5bfbec781f23569c1ec660" exitCode=0
Oct 30 20:59:59.954114 ip-10-0-137-72 kubenswrapper[2581]: I1030 20:59:59.953987    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerDied","Data":"e783eee8eb6fead7ab6db0e1267a635802dfa79afe5bfbec781f23569c1ec660"}
Oct 30 21:00:00.765427 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:00.765390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:00.765636 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:00.765525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:00.959690 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:00.959651    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" event={"ID":"8e300873-ed21-450b-9e6b-810ee569b783","Type":"ContainerStarted","Data":"15b21d8776e92253a2650e34906509e419075c6d46f0d5b2da523b077653dd36"}
Oct 30 21:00:01.032963 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.032863    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zkpxs" podStartSLOduration=6.796839025 podStartE2EDuration="17.032849272s" totalImagesPullingTime="10.236010247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.018583242 +0000 UTC m=+17.808736778" watchObservedRunningTime="2026-10-30 21:00:01.032849272 +0000 UTC m=+17.823002809"
Oct 30 21:00:01.468031 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.467780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:01.468184 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.468067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:01.468184 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.467925    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:01.468184 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.468168    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.468148372 +0000 UTC m=+34.258301903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:01.468352 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.468237    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:01.468352 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.468293    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.46827825 +0000 UTC m=+34.258431772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:01.569019 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.568932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:01.569156 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.569061    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:01.569156 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.569076    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:01.569156 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.569087    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:01.569156 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.569132    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.569121119 +0000 UTC m=+34.359274648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:01.765946 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.765905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:01.765946 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.765928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:01.766152 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.766019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:01.766152 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:01.766122    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:01.964215 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964135    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"ef416956f32ecd9f48ef9c4603a3fd9a9b8b77f5d6cba6dc3c16fc6246ae4385"}
Oct 30 21:00:01.964215 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964205    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"28a96efcc7d2a5d960c29f0a761b6da8739dd1465fb06a05f050c2b4f35a8097"}
Oct 30 21:00:01.964703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964234    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"1e28db969427611a9f83f6526cfc15efabc9d35f0acd668cafe43631279bd439"}
Oct 30 21:00:01.964703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964264    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"01341801f8b31e88f0f1e11bba1198caee8433a3230a3efd42f93076f4a700e6"}
Oct 30 21:00:01.964703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964293    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"ac592229ca7792189027a05f582e8e4feb85300814a2c213292de6a7220154e4"}
Oct 30 21:00:01.964703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:01.964319    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"e52208a56386c13660e49fd58d3fc589b776b89a1a74b892fbb00cf103d49129"}
Oct 30 21:00:02.765552 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:02.765502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:02.765736 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:02.765634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:03.766740 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:03.766655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:03.767258 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:03.766741    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:03.767258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:03.766816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:03.767258 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:03.766913    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:03.971398 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:03.971361    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"9735b6cceeb19dd0c6e3e3b7578c8b27176882b67e5272603f2456f22f38ced3"}
Oct 30 21:00:04.766138 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:04.766102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:04.766683 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:04.766639    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:05.766248 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:05.766132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:05.766248 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:05.766163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:05.766248 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:05.766230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:05.766953 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:05.766285    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:05.979451 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:05.979418    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" event={"ID":"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db","Type":"ContainerStarted","Data":"c7ea952f4fc8f13bf76bcd6c0d89ebc372eeb9f32202065e2a75affab048608f"}
Oct 30 21:00:05.979668 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:05.979650    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:05.995343 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:05.995319    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:06.069030 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:06.068932    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb" podStartSLOduration=7.091274203 podStartE2EDuration="22.068919912s" totalImagesPullingTime="14.977645709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.055137861 +0000 UTC m=+22.845291397" watchObservedRunningTime="2026-10-30 21:00:06.068919912 +0000 UTC m=+22.859073487"
Oct 30 21:00:06.766065 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:06.766032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:06.766232 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:06.766174    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:06.981156 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:06.981122    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:00:06.981695 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:06.981676    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:06.996731 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:06.996703    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:07.579948 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.579867    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:07.582551 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.582528    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bqg5"]
Oct 30 21:00:07.582680 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.582627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:07.582746 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:07.582721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:07.585384 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.585361    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5gz6"]
Oct 30 21:00:07.585478 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.585453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:07.585582 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:07.585564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:07.585893 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.585868    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nnm9g"]
Oct 30 21:00:07.585989 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:07.585952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:07.586027 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:07.586012    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:09.765897 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:09.765863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:09.766312 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:09.765863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:09.766312 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:09.765973    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:09.766312 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:09.765870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:09.766312 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:09.766049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:09.766312 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:09.766153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:11.766103 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:11.766071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:11.766103 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:11.766103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:11.766602 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:11.766071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:11.766602 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:11.766216    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:11.766602 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:11.766326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:11.766602 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:11.766411    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:13.766236 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:13.766045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:13.766592 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:13.766121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:13.766592 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:13.766303    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:13.766592 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:13.766388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:13.766592 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:13.766147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:13.766592 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:13.766476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:14.853670 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:14.853644    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rh2wd_2b36017a-ec01-4979-a358-7198261684d7/dns-node-resolver/0.log"
Oct 30 21:00:15.732941 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.732897    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:15.733117 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.733087    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:00:15.733698 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.733671    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:15.765698 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.765669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:15.765841 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.765731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:15.765841 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.765748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:15.765841 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:15.765822    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:15.765993 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:15.765934    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:15.766060 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:15.766039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:15.796789 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:15.796762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfthv_c56952e8-b525-401d-8128-e4c8ac0d6469/node-ca/0.log"
Oct 30 21:00:17.478928 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.478897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:17.478928 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.478931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:17.479457 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.479047    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:17.479457 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.479105    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs podName:1f1ff389-d912-4771-8a8f-43b6ab842829 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.479091906 +0000 UTC m=+66.269245435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs") pod "network-metrics-daemon-7bqg5" (UID: "1f1ff389-d912-4771-8a8f-43b6ab842829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:17.479457 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.479047    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:17.479457 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.479172    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret podName:0bbf8b47-e869-43e4-8fdd-459639f068c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.479161454 +0000 UTC m=+66.269314976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret") pod "global-pull-secret-syncer-n5gz6" (UID: "0bbf8b47-e869-43e4-8fdd-459639f068c3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:17.553289 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.553249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:17.580250 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.580218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:17.580413 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.580335    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:17.580413 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.580351    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:17.580413 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.580359    2581 projected.go:196] Error preparing data for projected volume kube-api-access-qmzhk for pod openshift-network-diagnostics/network-check-target-nnm9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:17.580413 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.580406    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk podName:9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.580394259 +0000 UTC m=+66.370547778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmzhk" (UniqueName: "kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk") pod "network-check-target-nnm9g" (UID: "9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:17.693404 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.693368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:17.765706 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.765669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:17.765873 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.765774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:17.765873 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.765794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:17.765873 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.765848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:17.766045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.765884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:17.766045 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:17.765961    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:17.901416 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:17.901335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:18.103510 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:18.103460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:18.307811 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:18.307733    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:19.765555 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:19.765520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:19.765555 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:19.765533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:19.766058 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:19.765652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:19.766058 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:19.765656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:19.766058 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:19.765772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:19.766058 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:19.765873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:21.766173 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:21.766141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:21.766173 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:21.766142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:21.766638 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:21.766242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:21.766638 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:21.766247    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:21.766638 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:21.766320    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:21.766638 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:21.766426    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:23.766328 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:23.766261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:23.766828 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:23.766408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:23.766828 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:23.766434    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:23.766828 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:23.766467    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:23.766828 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:23.766519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:23.766828 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:23.766588    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:25.765571 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:25.765535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:25.765971 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:25.765550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:25.765971 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:25.765649    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:25.765971 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:25.765550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:25.765971 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:25.765718    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:25.765971 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:25.765786    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:27.766221 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:27.766186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:27.766760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:27.766196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:27.766760 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:27.766298    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:27.766760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:27.766202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:27.766760 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:27.766389    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:27.766760 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:27.766453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:29.765910 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:29.765871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:29.765910 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:29.765900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:29.765910 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:29.765909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:29.766517 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:29.765999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:29.766517 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:29.766117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:29.766517 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:29.766208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:31.765659 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:31.765621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:31.766044 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:31.765621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:31.766044 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:31.765745    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bqg5" podUID="1f1ff389-d912-4771-8a8f-43b6ab842829"
Oct 30 21:00:31.766044 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:31.765829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gz6" podUID="0bbf8b47-e869-43e4-8fdd-459639f068c3"
Oct 30 21:00:31.766044 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:31.765621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:31.766044 ip-10-0-137-72 kubenswrapper[2581]: E1030 21:00:31.765930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nnm9g" podUID="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4"
Oct 30 21:00:31.984341 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:31.984270    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeReady"
Oct 30 21:00:31.984478 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:31.984383    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:00:32.044743 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.044703    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9lr5s"]
Oct 30 21:00:32.046560 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.046538    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zztpp"]
Oct 30 21:00:32.046697 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.046653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.048392 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.048371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.068605 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.068581    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4n77d"]
Oct 30 21:00:32.070196 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.070182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.072743 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.072723    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:00:32.074411 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.074392    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:00:32.074550 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.074533    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:00:32.075217 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.075197    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5jdx7\""
Oct 30 21:00:32.075399 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.075213    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:00:32.075399 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.075324    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l57v4\""
Oct 30 21:00:32.081168 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.081145    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:00:32.084963 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.084941    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9lr5s"]
Oct 30 21:00:32.087426 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.087405    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zztpp"]
Oct 30 21:00:32.089428 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.089410    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4n77d"]
Oct 30 21:00:32.172170 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.172138    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:00:32.172170 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.172144    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:00:32.172364 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.172249    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:00:32.174858 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.174847    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:00:32.175021 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.175009    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xstbj\""
Oct 30 21:00:32.182574 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxd4\" (UniqueName: \"kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.182668 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.182668 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.182734 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ntk\" (UniqueName: \"kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.182766 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.182797 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.182841 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dp2\" (UniqueName: \"kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.182876 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.182876 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.182934 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.182934 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.182895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284008 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.283920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284008 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.283960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ntk\" (UniqueName: \"kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284008 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.283981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dp2\" (UniqueName: \"kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284344 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284678 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284480    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.284678 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxd4\" (UniqueName: \"kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.284678 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284641    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284776 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.284889 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.284869    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.285063 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.285047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.287827 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.287801    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.288045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.288022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.288137 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.288123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.304714 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.304688    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxd4\" (UniqueName: \"kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.305153 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.305129    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dp2\" (UniqueName: \"kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.305231 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.305155    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ntk\" (UniqueName: \"kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.361803 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.361768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:32.367365 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.367344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:32.382252 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.382227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:32.677936 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.677908    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9lr5s"]
Oct 30 21:00:32.680118 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.680096    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zztpp"]
Oct 30 21:00:32.681061 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:32.681038    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ea0220_eb35_4f3d_8147_060179390f3d.slice/crio-839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3 WatchSource:0}: Error finding container 839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3: Status 404 returned error can't find the container with id 839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3
Oct 30 21:00:32.707848 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:32.707828    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4n77d"]
Oct 30 21:00:32.708925 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:32.708901    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb088453c_eec1_465f_939e_e4c78cc2290b.slice/crio-59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb WatchSource:0}: Error finding container 59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb: Status 404 returned error can't find the container with id 59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb
Oct 30 21:00:33.036388 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.036313    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4n77d" event={"ID":"b088453c-eec1-465f-939e-e4c78cc2290b","Type":"ContainerStarted","Data":"59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb"}
Oct 30 21:00:33.037924 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.037896    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zztpp" event={"ID":"c2ea0220-eb35-4f3d-8147-060179390f3d","Type":"ContainerStarted","Data":"44430e2703c1e0add0d67d0ae15911115e0458e0ed7bafaf25108d3bea939c33"}
Oct 30 21:00:33.038054 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.037945    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zztpp" event={"ID":"c2ea0220-eb35-4f3d-8147-060179390f3d","Type":"ContainerStarted","Data":"839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3"}
Oct 30 21:00:33.038923 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.038901    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lr5s" event={"ID":"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c","Type":"ContainerStarted","Data":"7f5fc37f005cd26e28357b4f0e46e6f64b67c18191c1d762048948f1316d9f2b"}
Oct 30 21:00:33.570296 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.570203    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rh2wd_2b36017a-ec01-4979-a358-7198261684d7/dns-node-resolver/0.log"
Oct 30 21:00:33.767862 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.766866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:33.767862 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.767206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:33.767862 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.767462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:33.784126 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.783692    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:00:33.784126 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.783742    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:00:33.797175 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.797151    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fz2bf\""
Oct 30 21:00:33.797357 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.797332    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jljmv\""
Oct 30 21:00:33.797457 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.797424    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:00:33.797526 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.797458    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:00:33.797526 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:33.797475    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-56hzc\""
Oct 30 21:00:34.045276 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:34.045191    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zztpp" event={"ID":"c2ea0220-eb35-4f3d-8147-060179390f3d","Type":"ContainerStarted","Data":"ce848f0d5cc1c38aea3bfcde56fadb0474c985b2a739d72d77c7e27d40497e7d"}
Oct 30 21:00:34.769577 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:34.769514    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfthv_c56952e8-b525-401d-8128-e4c8ac0d6469/node-ca/0.log"
Oct 30 21:00:35.050911 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.050824    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zztpp" event={"ID":"c2ea0220-eb35-4f3d-8147-060179390f3d","Type":"ContainerStarted","Data":"50f031fe94a1008d328b074ae26561e22bb94a00f72bfda1bd7bcd92ed0cea37"}
Oct 30 21:00:35.053107 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.053080    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lr5s" event={"ID":"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c","Type":"ContainerStarted","Data":"f3b4f1b7276fdb0263dae4a14b9146d2f71968c58dbb8469f6d3319374556abf"}
Oct 30 21:00:35.053235 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.053126    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lr5s" event={"ID":"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c","Type":"ContainerStarted","Data":"3a58cca7737a6872b06203597c0025095b2fe10748fe8f2398d4330c9eb96a31"}
Oct 30 21:00:35.053235 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.053197    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:35.150695 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.150644    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zztpp" podStartSLOduration=1.195324291 podStartE2EDuration="3.15063082s" totalImagesPullingTime="1.955306529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:35.107662703 +0000 UTC m=+51.897816244" watchObservedRunningTime="2026-10-30 21:00:35.15063082 +0000 UTC m=+51.940784357"
Oct 30 21:00:35.150866 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.150851    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9lr5s" podStartSLOduration=1.3838732519999999 podStartE2EDuration="3.150844554s" totalImagesPullingTime="1.766971302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:35.150505735 +0000 UTC m=+51.940659272" watchObservedRunningTime="2026-10-30 21:00:35.150844554 +0000 UTC m=+51.940998090"
Oct 30 21:00:35.456355 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.456317    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"]
Oct 30 21:00:35.458199 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.458183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.474690 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.474667    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:00:35.474887 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.474871    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:00:35.474948 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.474916    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:00:35.474948 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.474929    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:00:35.477476 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.477449    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s9xpp\""
Oct 30 21:00:35.488615 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.488591    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"]
Oct 30 21:00:35.514517 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.514483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.514626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.514529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm74\" (UniqueName: \"kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.514626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.514556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.514626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.514605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.575446 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.575385    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:00:35.615398 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.615365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm74\" (UniqueName: \"kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.615583 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.615404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.615583 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.615430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.615583 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.615565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.616132 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.616109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.617854 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.617834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.617930 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.617904    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.635927 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.635901    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm74\" (UniqueName: \"kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:35.770580 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:35.770555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:36.057837 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.057739    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4n77d" event={"ID":"b088453c-eec1-465f-939e-e4c78cc2290b","Type":"ContainerStarted","Data":"ec3a5ca68e336db665d0879bc369f4e97f957b360d49156a3a405be1393b20e5"}
Oct 30 21:00:36.061771 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.060873    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"]
Oct 30 21:00:36.065179 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:36.065153    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e040ec_3520_40f4_b906_cc22de4d532a.slice/crio-dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c WatchSource:0}: Error finding container dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c: Status 404 returned error can't find the container with id dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c
Oct 30 21:00:36.100069 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.100020    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4n77d" podStartSLOduration=1.488412896 podStartE2EDuration="4.100006698s" totalImagesPullingTime="2.611593802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:36.099331888 +0000 UTC m=+52.889485424" watchObservedRunningTime="2026-10-30 21:00:36.100006698 +0000 UTC m=+52.890160234"
Oct 30 21:00:36.807437 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.807403    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:00:36.809371 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.809349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823640 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823881 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.823933 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.823910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.824047 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.824005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.824108 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.824062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.825484 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.825462    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:00:36.825484 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.825467    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:00:36.838933 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.838913    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:00:36.838933 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.838911    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:00:36.839090 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.838962    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:00:36.839090 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.839067    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:00:36.839159 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.839141    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:00:36.839261 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.839244    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jqkvq\""
Oct 30 21:00:36.867454 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.867430    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:00:36.868962 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.868940    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:00:36.924715 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.924687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.924715 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.924718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.924939 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.924765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925260 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925234    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925370 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925464 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925464 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925464 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.925640 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.925528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.926043 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.926011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.926153 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.926136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.926235 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.926215    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.926306 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.926286    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.927743 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.927722    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.927978 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.927956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:36.946403 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:36.946364    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:37.062310 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:37.062224    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z" event={"ID":"87e040ec-3520-40f4-b906-cc22de4d532a","Type":"ContainerStarted","Data":"dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c"}
Oct 30 21:00:37.121409 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:37.121381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:37.429998 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:37.429976    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:00:37.431263 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:37.431242    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23412f3_007f_4358_a2a1_09288ff60856.slice/crio-38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300 WatchSource:0}: Error finding container 38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300: Status 404 returned error can't find the container with id 38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300
Oct 30 21:00:38.069234 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:38.069198    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z" event={"ID":"87e040ec-3520-40f4-b906-cc22de4d532a","Type":"ContainerStarted","Data":"7500c5f87cc9ab98aa70d78c9fa9dbbf4fb61ba8da889da2efa8ab4fcb351dfa"}
Oct 30 21:00:38.069718 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:38.069259    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z" event={"ID":"87e040ec-3520-40f4-b906-cc22de4d532a","Type":"ContainerStarted","Data":"638c37f781ee2f6a677a7a01ba22642d9ff81fb17d370144e0bce4be77950ae8"}
Oct 30 21:00:38.070655 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:38.070634    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbc659c8d-gwbfw" event={"ID":"b23412f3-007f-4358-a2a1-09288ff60856","Type":"ContainerStarted","Data":"38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300"}
Oct 30 21:00:38.115319 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:38.115238    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z" podStartSLOduration=1.826842157 podStartE2EDuration="3.115218031s" totalImagesPullingTime="1.288375874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:38.113331804 +0000 UTC m=+54.903485364" watchObservedRunningTime="2026-10-30 21:00:38.115218031 +0000 UTC m=+54.905371569"
Oct 30 21:00:38.997337 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:38.997303    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:40.015383 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.015348    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"]
Oct 30 21:00:40.019090 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.019071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.038559 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.038536    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:00:40.038559 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.038532    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:00:40.039248 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.039227    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"]
Oct 30 21:00:40.048879 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.048855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.048974 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.048886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5v2k\" (UniqueName: \"kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.048974 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.048909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.049066 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.049042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.051799 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.051780    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6gs8c\""
Oct 30 21:00:40.052986 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.052969    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"]
Oct 30 21:00:40.055698 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.055683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.063624 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.063606    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jdnjm"]
Oct 30 21:00:40.066654 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.066636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.073779 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.073761    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:00:40.080724 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.080702    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:00:40.080828 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.080749    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:00:40.080828 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.080801    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jxvr7\""
Oct 30 21:00:40.081362 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.081339    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"]
Oct 30 21:00:40.138750 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.138723    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-595gg\""
Oct 30 21:00:40.138899 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.138882    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:00:40.139099 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.139083    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.149883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.149925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.149953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.149980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150067 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.150418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.150418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5v2k\" (UniqueName: \"kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.150418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.150418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk8r\" (UniqueName: \"kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150605 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.150605 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.150605 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.150721 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.150721 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgh27\" (UniqueName: \"kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.150721 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.150861 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.151053 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.150915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.151053 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.151030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.153208 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.153180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.153310 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.153209    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.171547 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.171517    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5v2k\" (UniqueName: \"kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.173870 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.173842    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:00:40.252273 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252273 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252456 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252456 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252456 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252438    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252594 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252594 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252594 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252520    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252594 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk8r\" (UniqueName: \"kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252760 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgh27\" (UniqueName: \"kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252790    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.252993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252893    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.252993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.252938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.253260 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.253098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.253564 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.253340    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.253564 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.253483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.253564 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.253525    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.253726 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.253608    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.254900 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.254865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.254999 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.254939    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.255583 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.255562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.255634 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.255603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.273967 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.273913    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk8r\" (UniqueName: \"kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.274103 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.274086    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgh27\" (UniqueName: \"kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.334928 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.334898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:40.371123 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.371087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:40.383826 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.383798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:40.642837 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.642815    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"]
Oct 30 21:00:40.644219 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:40.644191    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8062ab8_4d4e_41a1_9ed0_1328ffc23870.slice/crio-934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c WatchSource:0}: Error finding container 934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c: Status 404 returned error can't find the container with id 934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c
Oct 30 21:00:40.674983 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:40.674958    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"]
Oct 30 21:00:40.676169 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:40.676148    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548cadb6_b5c0_4f7d_ad73_e5060de08fc9.slice/crio-8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c WatchSource:0}: Error finding container 8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c: Status 404 returned error can't find the container with id 8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c
Oct 30 21:00:41.083128 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.083081    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87" event={"ID":"e8062ab8-4d4e-41a1-9ed0-1328ffc23870","Type":"ContainerStarted","Data":"ecb4aa0fd41f428bd229772152e796643d71d3ab4d6cd506c55d03081ff018fd"}
Oct 30 21:00:41.084893 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.083156    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87" event={"ID":"e8062ab8-4d4e-41a1-9ed0-1328ffc23870","Type":"ContainerStarted","Data":"6b67b32a87a692c7900798dda1ffdf2d999a8da0d38b6c83a7171a119d3b063a"}
Oct 30 21:00:41.084893 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.083177    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87" event={"ID":"e8062ab8-4d4e-41a1-9ed0-1328ffc23870","Type":"ContainerStarted","Data":"934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c"}
Oct 30 21:00:41.084893 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.084630    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jdnjm" event={"ID":"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19","Type":"ContainerStarted","Data":"381d873ef40bf89608a47b7892c820ef3c65254b998e32b886c5f453788d4445"}
Oct 30 21:00:41.085999 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.085960    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m" event={"ID":"548cadb6-b5c0-4f7d-ad73-e5060de08fc9","Type":"ContainerStarted","Data":"8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c"}
Oct 30 21:00:41.088110 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.088084    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbc659c8d-gwbfw" event={"ID":"b23412f3-007f-4358-a2a1-09288ff60856","Type":"ContainerStarted","Data":"ca4a09be5ce9c1a71ef99b2b19b8ada98743046220fc6115bb39c7c087f59a0b"}
Oct 30 21:00:41.130520 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.130457    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fbc659c8d-gwbfw" podStartSLOduration=2.490700341 podStartE2EDuration="5.130440479s" totalImagesPullingTime="2.639740138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:41.1300227 +0000 UTC m=+57.920176239" watchObservedRunningTime="2026-10-30 21:00:41.130440479 +0000 UTC m=+57.920594015"
Oct 30 21:00:41.398772 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.398731    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:00:41.409028 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.407791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.425764 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.425740    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:00:41.454291 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.454259    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:00:41.454446 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.454375    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:00:41.456179 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.456079    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:00:41.466218 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466348 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466348 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466348 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466348 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466845 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.466845 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.466641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.525397 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525365    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:00:41.525552 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525413    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:00:41.525707 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525687    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:00:41.525825 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525724    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:00:41.525900 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525848    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:00:41.525900 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525872    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:00:41.525968 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.525956    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lm7tg\""
Oct 30 21:00:41.567676 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.567839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.567839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.567839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.567839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.567980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.568032 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.568009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.569397 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.568041    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.569397 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.568051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.569397 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.568079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.569397 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.568829    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.569625 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.569435    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.571904 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.571853    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.572226 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.572201    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.572826 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.572798    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.573064 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.573044    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.573106 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.573077    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.573330 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.573303    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.573483 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.573464    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.573588 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.573529    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.574561 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.574540    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.589936 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.589878    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:41.738318 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:41.738281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:42.062544 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.062519    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:00:42.063458 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:42.063435    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b802282_8c04_4522_8d63_1d3b66d897aa.slice/crio-1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812 WatchSource:0}: Error finding container 1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812: Status 404 returned error can't find the container with id 1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812
Oct 30 21:00:42.101558 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.101515    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m" event={"ID":"548cadb6-b5c0-4f7d-ad73-e5060de08fc9","Type":"ContainerStarted","Data":"6b90f40df895245be3570ed7f9965fd7fe531c0f6118b376379cc346d13d1ca6"}
Oct 30 21:00:42.101895 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.101599    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m" event={"ID":"548cadb6-b5c0-4f7d-ad73-e5060de08fc9","Type":"ContainerStarted","Data":"d2048b31487aec461065af350e21a809d86e91c98dc8c097b4ed3df7f15382b3"}
Oct 30 21:00:42.102953 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.102929    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812"}
Oct 30 21:00:42.105842 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.105178    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" containerID="2951873accf1d2eb91d98be2a7c994f095fb00a3a565235b5c624a7eac06bb60" exitCode=0
Oct 30 21:00:42.105842 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.105770    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jdnjm" event={"ID":"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19","Type":"ContainerDied","Data":"2951873accf1d2eb91d98be2a7c994f095fb00a3a565235b5c624a7eac06bb60"}
Oct 30 21:00:42.233437 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.233409    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"]
Oct 30 21:00:42.239657 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.238792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.259711 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.259281    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"]
Oct 30 21:00:42.260453 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260436    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260638    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260675    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2nq7z\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260746    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260746    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260824    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-druc28d9l6vvr\""
Oct 30 21:00:42.262556 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.260835    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:00:42.275671 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.275778 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.275778 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.275778 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.276027 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.276027 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bkz\" (UniqueName: \"kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.276027 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.275998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.276169 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.276037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.276169 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.276070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376693 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376795 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376795 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376795 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376795 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bkz\" (UniqueName: \"kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.376993 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.376895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.377538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.377450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.377762 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.377739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.380242 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.379820    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.380242 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.380096    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.380242 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.380207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.380375 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.380280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.380462 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.380437    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.381121 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.381098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.397265 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.397241    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bkz\" (UniqueName: \"kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.578134 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.578097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:42.892034 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:42.892002    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"]
Oct 30 21:00:42.893568 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:42.893538    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943e6093_44ff_4e3e_97d4_67a24ce560a0.slice/crio-1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da WatchSource:0}: Error finding container 1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da: Status 404 returned error can't find the container with id 1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da
Oct 30 21:00:43.111068 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.111022    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87" event={"ID":"e8062ab8-4d4e-41a1-9ed0-1328ffc23870","Type":"ContainerStarted","Data":"5c691d825260b9a357d494f8fff34692e5521c3ec5b3d38e69ada69096d0c5ca"}
Oct 30 21:00:43.112256 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.112231    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da"}
Oct 30 21:00:43.114158 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.114132    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jdnjm" event={"ID":"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19","Type":"ContainerStarted","Data":"3327bb1a02789cd584bda8b263c260d5cc868897cab9adf2170d8cf01bb817ee"}
Oct 30 21:00:43.114237 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.114183    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jdnjm" event={"ID":"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19","Type":"ContainerStarted","Data":"09dcde14dfd608fd51c0112ffbe3890cbddad858c828857759f5432082792e08"}
Oct 30 21:00:43.116228 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.116207    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m" event={"ID":"548cadb6-b5c0-4f7d-ad73-e5060de08fc9","Type":"ContainerStarted","Data":"3da9ad21a2989f0b0c07a46ddade64c9b0f3b9ad7fd5e555a95ceb29b417b229"}
Oct 30 21:00:43.152793 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.152749    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87" podStartSLOduration=2.597234685 podStartE2EDuration="4.152735446s" totalImagesPullingTime="1.555500761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:43.151892245 +0000 UTC m=+59.942045781" watchObservedRunningTime="2026-10-30 21:00:43.152735446 +0000 UTC m=+59.942888982"
Oct 30 21:00:43.196949 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.196904    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m" podStartSLOduration=1.931193666 podStartE2EDuration="3.196891372s" totalImagesPullingTime="1.265697706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:43.196167055 +0000 UTC m=+59.986320592" watchObservedRunningTime="2026-10-30 21:00:43.196891372 +0000 UTC m=+59.987044907"
Oct 30 21:00:43.240405 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:43.240355    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jdnjm" podStartSLOduration=2.226249362 podStartE2EDuration="3.240340986s" totalImagesPullingTime="931.860031ms" totalInitContainerRuntime="82.231593ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:43.239846652 +0000 UTC m=+60.030000191" watchObservedRunningTime="2026-10-30 21:00:43.240340986 +0000 UTC m=+60.030494544"
Oct 30 21:00:44.121703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.121666    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="d0159d09e37f3f934914c9cd69bce4ae05309ea35e90c490c3329541151d95f3" exitCode=0
Oct 30 21:00:44.122109 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.121763    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"d0159d09e37f3f934914c9cd69bce4ae05309ea35e90c490c3329541151d95f3"}
Oct 30 21:00:44.598568 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.598522    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"]
Oct 30 21:00:44.601825 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.601799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.622404 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.622376    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:00:44.625634 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.625609    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:00:44.630410 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.630386    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"]
Oct 30 21:00:44.659125 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.659094    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:00:44.659125 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.659124    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pdr56\""
Oct 30 21:00:44.659341 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.659191    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5p0s1ifovnrri\""
Oct 30 21:00:44.659341 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.659284    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:00:44.700837 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.700805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701033 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.700854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxfr\" (UniqueName: \"kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701033 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.700914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701033 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.700957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701033 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.700981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701033 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.701024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.701268 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.701058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802009 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.801975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802131 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxfr\" (UniqueName: \"kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802131 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802131 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802294 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802294 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.802294 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.802217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.803305 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.803277    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.803558 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.803536    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.804775 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.804750    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.807305 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.807285    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.812626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.812598    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.814787 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.814749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.825946 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.825830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxfr\" (UniqueName: \"kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:44.891010 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.890874    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"]
Oct 30 21:00:44.896204 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.896177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:44.914167 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.914142    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"]
Oct 30 21:00:44.914842 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.914817    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q65n4\""
Oct 30 21:00:44.915116 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.915097    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:00:44.916537 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:44.916191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:45.005470 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.005208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.005470 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.005290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm67s\" (UniqueName: \"kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.061755 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.060980    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:45.107188 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.107152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.107357 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.107228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm67s\" (UniqueName: \"kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.117025 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.116940    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.143360 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.143327    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm67s\" (UniqueName: \"kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.152613 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.152282    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"461def1e391bb7f08aea6b81b86c80b51e86d4646cb32a7961226803dc346049"}
Oct 30 21:00:45.152613 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.152448    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"f41194b9e84f9a09285f2b240f33b664c15d93367c718eff963edf7534e6be9d"}
Oct 30 21:00:45.152613 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.152539    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"4f69feaa53436c59164a555653b36dc0cf236c091ff8f834e1bf6c86f4301ba7"}
Oct 30 21:00:45.211196 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.211164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:45.237878 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.237850    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"]
Oct 30 21:00:45.240572 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:45.240528    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce43bf79_f468_4826_ae78_e58f103f9fa7.slice/crio-bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b WatchSource:0}: Error finding container bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b: Status 404 returned error can't find the container with id bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b
Oct 30 21:00:45.275099 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.272752    2581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:00:45.311695 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.311668    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:00:45.316846 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.316820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.343767 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.343734    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:00:45.410340 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410340 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410574 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410574 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410670 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410670 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410741 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.410790 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.410749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512079 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512506 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512506 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512820 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512799    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.512921 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512894    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.513024 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512921    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.513024 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.512961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.513245 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.513179    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.514918 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.514894    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.515040 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.514992    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.516659 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.516637    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"]
Oct 30 21:00:45.532715 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.532693    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.534648 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:45.534627    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8211ee52_aa1e_4a0c_bfe8_3e1779da22a7.slice/crio-9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b WatchSource:0}: Error finding container 9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b: Status 404 returned error can't find the container with id 9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b
Oct 30 21:00:45.631509 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.631454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:45.944780 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:45.944674    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:00:45.957027 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:45.956954    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826b8c4d_26f6_47f8_a885_85c93b1996cf.slice/crio-8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68 WatchSource:0}: Error finding container 8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68: Status 404 returned error can't find the container with id 8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68
Oct 30 21:00:46.158242 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.158204    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc" event={"ID":"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7","Type":"ContainerStarted","Data":"9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b"}
Oct 30 21:00:46.164045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.163773    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"b10115285c74c45ca93512c5fd3b33191df94e749aa989979852a32d01b18984"}
Oct 30 21:00:46.164045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.163861    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"7f9524e0428edc86db935ed9ec407b92bb603a968adfffbc86c5d49d90ffacdf"}
Oct 30 21:00:46.164045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.163903    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"d8789f8b84d716526cc97428b87cbcf6be6067030687d76f4c49ae4023623c89"}
Oct 30 21:00:46.164045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.163947    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"9b0e0a81703b0ccbdc596b2ff370db2913e1364f033d243f6059c890cbf5b61f"}
Oct 30 21:00:46.164045 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.163984    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"850fb538bf612204d79d236a210b091274501637dc3ddccc4417bafa620a6015"}
Oct 30 21:00:46.166399 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.166367    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779db79f49-fqpnf" event={"ID":"826b8c4d-26f6-47f8-a885-85c93b1996cf","Type":"ContainerStarted","Data":"d41f6145e7fa093d953c3c92d6af819e0d462f2067d2e4d5ff35972bcfbc2caf"}
Oct 30 21:00:46.166557 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.166419    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779db79f49-fqpnf" event={"ID":"826b8c4d-26f6-47f8-a885-85c93b1996cf","Type":"ContainerStarted","Data":"8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68"}
Oct 30 21:00:46.168614 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.168587    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj" event={"ID":"ce43bf79-f468-4826-ae78-e58f103f9fa7","Type":"ContainerStarted","Data":"bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b"}
Oct 30 21:00:46.226058 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.225960    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-779db79f49-fqpnf" podStartSLOduration=1.225943082 podStartE2EDuration="1.225943082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:46.225066757 +0000 UTC m=+63.015220296" watchObservedRunningTime="2026-10-30 21:00:46.225943082 +0000 UTC m=+63.016096619"
Oct 30 21:00:46.999028 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:46.999000    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:00:47.004822 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.004793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.022577 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.022549    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:00:47.022839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.022815    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:00:47.022839 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.022827    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:00:47.022997 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.022954    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:00:47.023049 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.022996    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:00:47.023049 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.023035    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:00:47.023049 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.023036    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vwnx\""
Oct 30 21:00:47.023201 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.023079    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1c035iq9e8bv\""
Oct 30 21:00:47.027279 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.027254    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:00:47.027399 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.027352    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:00:47.027951 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.027932    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:00:47.033176 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.028019    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:00:47.033176 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.029265    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:00:47.041735 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.041620    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:00:47.121997 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.121959    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:47.122788 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.122760    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:00:47.131332 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131435 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131528 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131528 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131594 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131654 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131700 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131754 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131847 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131847 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131912 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131912 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131991 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.131991 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.131971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.132062 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.132000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.132062 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.132024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.132062 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.132054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.132170 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.132077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.138086 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.138058    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:00:47.138219 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.138064    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:00:47.177102 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.177010    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerStarted","Data":"6224f1d7cb08e8039afc519069589f90c1de09e4bd3e77d67fac145de4b34f06"}
Oct 30 21:00:47.181258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.181180    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"d3b95e60ad8bf7cbd35087c8719d515269c8bc8099382623c51524e23d21a6e2"}
Oct 30 21:00:47.181258 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.181249    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"6abbf1a4351663253c090174ccf319cad0a34b911122091013e20266ff88c283"}
Oct 30 21:00:47.181410 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.181288    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" event={"ID":"943e6093-44ff-4e3e-97d4-67a24ce560a0","Type":"ContainerStarted","Data":"c43cd169e37c293cc59d7b3d6c648da4ec89dc09d31ae11d17756914b11b0d0b"}
Oct 30 21:00:47.181410 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.181339    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:47.182996 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.182974    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj" event={"ID":"ce43bf79-f468-4826-ae78-e58f103f9fa7","Type":"ContainerStarted","Data":"c254292f87ae8e05757930c5c1a8522517e845f07231d0e8d16fc518fdd84211"}
Oct 30 21:00:47.231291 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.231242    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.421263185 podStartE2EDuration="6.231228474s" totalImagesPullingTime="2.801268357s" totalInitContainerRuntime="8.696932ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:47.230520483 +0000 UTC m=+64.020674022" watchObservedRunningTime="2026-10-30 21:00:47.231228474 +0000 UTC m=+64.021382011"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.232944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.233538 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233641    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233659    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.233714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.234094    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234350 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.234115    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234867 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.234801    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.234968 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.234946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.236026 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.235999    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.239002 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.237319    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.239586 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.239565    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.239719 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.238560    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.240170 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.240148    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.241012 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.240987    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.241720 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.241700    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.241880 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.241856    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.241880 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.241750    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.243384 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.243356    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.243482 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.243393    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.243569 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.243522    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.244162 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.244122    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.256568 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.256541    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.325576 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.325517    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" podStartSLOduration=2.214151822 podStartE2EDuration="5.325479069s" totalImagesPullingTime="3.111327247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:47.280631269 +0000 UTC m=+64.070784805" watchObservedRunningTime="2026-10-30 21:00:47.325479069 +0000 UTC m=+64.115632609"
Oct 30 21:00:47.325935 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.325906    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj" podStartSLOduration=1.6378166539999999 podStartE2EDuration="3.325897339s" totalImagesPullingTime="1.688080685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:47.324416264 +0000 UTC m=+64.114569802" watchObservedRunningTime="2026-10-30 21:00:47.325897339 +0000 UTC m=+64.116050878"
Oct 30 21:00:47.349386 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.349357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:47.677408 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:47.677387    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:00:47.678152 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:47.678125    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453a3396_2bff_41c7_8164_25a8991c4e2d.slice/crio-2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829 WatchSource:0}: Error finding container 2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829: Status 404 returned error can't find the container with id 2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829
Oct 30 21:00:48.188703 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.188666    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc" event={"ID":"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7","Type":"ContainerStarted","Data":"6fc256ffd2b64feb34a773bc953dcb16e7fb6720993f90257e603112ee3b482d"}
Oct 30 21:00:48.189183 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.188885    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:48.190515 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.190474    2581 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699" exitCode=0
Oct 30 21:00:48.190626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.190519    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"}
Oct 30 21:00:48.190626 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.190584    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829"}
Oct 30 21:00:48.194828 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.194811    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:48.234234 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:48.234185    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc" podStartSLOduration=2.4686369519999998 podStartE2EDuration="4.234173132s" totalImagesPullingTime="1.76553618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:48.233920183 +0000 UTC m=+65.024073720" watchObservedRunningTime="2026-10-30 21:00:48.234173132 +0000 UTC m=+65.024326668"
Oct 30 21:00:49.559683 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.559642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:49.560166 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.559771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:49.575166 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.575138    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:00:49.582898 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.582863    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:49.588617 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.588584    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:00:49.593100 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.593073    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:49.660352 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.660315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:49.676051 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.676010    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:00:49.696358 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.696327    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:00:49.699415 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.699389    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-56hzc\""
Oct 30 21:00:49.704415 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.704313    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:49.704415 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.704386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:49.722883 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.722862    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fz2bf\""
Oct 30 21:00:49.728010 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:49.727986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:50.009407 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.009380    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5gz6"]
Oct 30 21:00:50.009854 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:50.009825    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbf8b47_e869_43e4_8fdd_459639f068c3.slice/crio-1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef WatchSource:0}: Error finding container 1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef: Status 404 returned error can't find the container with id 1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef
Oct 30 21:00:50.014115 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.014093    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jljmv\""
Oct 30 21:00:50.017213 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.017192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:50.040053 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:50.040025    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1ff389_d912_4771_8a8f_43b6ab842829.slice/crio-6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b WatchSource:0}: Error finding container 6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b: Status 404 returned error can't find the container with id 6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b
Oct 30 21:00:50.040422 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.040399    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bqg5"]
Oct 30 21:00:50.200978 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.200941    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bqg5" event={"ID":"1f1ff389-d912-4771-8a8f-43b6ab842829","Type":"ContainerStarted","Data":"6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b"}
Oct 30 21:00:50.202483 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.202451    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n5gz6" event={"ID":"0bbf8b47-e869-43e4-8fdd-459639f068c3","Type":"ContainerStarted","Data":"1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef"}
Oct 30 21:00:50.313678 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:50.313653    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nnm9g"]
Oct 30 21:00:50.314835 ip-10-0-137-72 kubenswrapper[2581]: W1030 21:00:50.314802    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cabdea7_af0c_4c28_a3e4_d2ae7cce80d4.slice/crio-0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29 WatchSource:0}: Error finding container 0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29: Status 404 returned error can't find the container with id 0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29
Oct 30 21:00:51.208907 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:51.208863    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nnm9g" event={"ID":"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4","Type":"ContainerStarted","Data":"0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29"}
Oct 30 21:00:52.231569 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.231255    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"}
Oct 30 21:00:52.231569 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.231346    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"}
Oct 30 21:00:52.231569 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.231383    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"}
Oct 30 21:00:52.231569 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.231419    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"}
Oct 30 21:00:52.235457 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.234651    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bqg5" event={"ID":"1f1ff389-d912-4771-8a8f-43b6ab842829","Type":"ContainerStarted","Data":"01b27cb690749b358c78ab1b3a7e5bd6f4db444f0150d0611424189a53917c7b"}
Oct 30 21:00:52.235457 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.234708    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bqg5" event={"ID":"1f1ff389-d912-4771-8a8f-43b6ab842829","Type":"ContainerStarted","Data":"5a40c22938f7f8cbdb8e2aec7fdee03528d85f5ae2b74b003a8947ee0741d064"}
Oct 30 21:00:52.287332 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:52.286882    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7bqg5" podStartSLOduration=67.185310356 podStartE2EDuration="1m8.286857715s" totalImagesPullingTime="1.101547359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:52.283222864 +0000 UTC m=+69.073376401" watchObservedRunningTime="2026-10-30 21:00:52.286857715 +0000 UTC m=+69.077011253"
Oct 30 21:00:53.203335 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:53.201988    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:53.250615 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:53.250580    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"}
Oct 30 21:00:53.251093 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:53.250663    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerStarted","Data":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"}
Oct 30 21:00:53.322657 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:53.322594    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.784218481 podStartE2EDuration="7.32257279s" totalImagesPullingTime="3.517569533s" totalInitContainerRuntime="20.784776ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:53.319461759 +0000 UTC m=+70.109615296" watchObservedRunningTime="2026-10-30 21:00:53.32257279 +0000 UTC m=+70.112726328"
Oct 30 21:00:55.275709 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.275622    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nnm9g" event={"ID":"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4","Type":"ContainerStarted","Data":"3db1d2702e5f6dd52b007e61d688d8c859f0b81432f1aa734898fb7c2a834d98"}
Oct 30 21:00:55.276167 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.275764    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:55.317242 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.317186    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nnm9g" podStartSLOduration=66.678360608 podStartE2EDuration="1m11.317170372s" totalImagesPullingTime="4.638809764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:55.31583155 +0000 UTC m=+72.105985088" watchObservedRunningTime="2026-10-30 21:00:55.317170372 +0000 UTC m=+72.107323909"
Oct 30 21:00:55.632418 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.632384    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:55.632641 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.632452    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:55.638823 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:55.638791    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:56.285751 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:56.285723    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:56.530368 ip-10-0-137-72 kubenswrapper[2581]: I1030 21:00:56.530217    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:56.530401 ip-10-0-137-72 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:00:56.541889 ip-10-0-137-72 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:00:56.542058 ip-10-0-137-72 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:00:56.542106 ip-10-0-137-72 systemd[1]: kubelet.service: Consumed 4.677s CPU time.
Oct 30 21:00:56.557235 ip-10-0-137-72 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:00:56.617462 ip-10-0-137-72 kubenswrapper[7037]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:56.617462 ip-10-0-137-72 kubenswrapper[7037]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:00:56.617462 ip-10-0-137-72 kubenswrapper[7037]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:56.617462 ip-10-0-137-72 kubenswrapper[7037]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:56.619533 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619440    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.619533 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619533    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619537    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619540    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619544    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619547    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619550    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619553    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619555    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619558    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619561    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619564    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619567    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619578    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619581    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619584    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619587    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619590    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619593    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619596    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619598    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.619602 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619601    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619604    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619607    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619610    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619613    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619616    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619619    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619623    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619628    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619632    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619635    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619639    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619642    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619645    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619647    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619650    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619653    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619655    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619658    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.620066 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619661    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619663    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619666    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619668    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619671    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619674    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619680    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619683    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619685    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619688    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619690    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619693    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619698    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619701    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619703    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619706    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619709    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619712    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619714    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619717    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.620568 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619719    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619722    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619725    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619728    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619731    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619733    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619736    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619739    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619741    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619744    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619746    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619749    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619752    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619755    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619757    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619760    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619763    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619765    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619768    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.621041 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619771    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619774    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619776    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619779    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619782    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619788    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619791    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619794    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619797    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619799    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619802    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619805    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619807    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619810    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619812    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619815    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619818    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619821    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619824    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619827    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.621483 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619829    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619832    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619835    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619837    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619840    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619843    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.619845    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620631    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620640    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620644    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620647    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620650    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620653    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620656    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620659    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620663    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620667    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620671    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620674    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.621973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620677    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620680    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620682    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620685    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620688    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620690    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620693    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620696    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620698    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620701    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620704    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620707    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620710    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620713    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620715    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620718    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620721    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620723    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620726    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620728    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.622425 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620731    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620734    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620736    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620739    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620742    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620744    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620747    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620750    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620753    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620756    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620758    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620762    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620765    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620767    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620770    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620772    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620775    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620778    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620780    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.622910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620783    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620786    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620788    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620791    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620793    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620796    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620799    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620801    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620804    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620806    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620809    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620811    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620814    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620816    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620819    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620823    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620827    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620829    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620832    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.623348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620836    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620838    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620842    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620845    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620847    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620850    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620853    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620856    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620858    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620861    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620863    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620866    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620869    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620872    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620874    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620877    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620879    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620882    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620885    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620887    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.623833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620890    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620892    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620895    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620898    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620900    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620903    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620906    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620908    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620911    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620913    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620919    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620922    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620925    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620927    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620930    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.620934    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621403    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621409    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621413    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621416    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.624321 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621420    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621423    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621426    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621429    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621431    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621434    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621437    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621440    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621443    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621445    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621448    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621450    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621453    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621456    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621458    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621461    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621464    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621467    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621469    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621472    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.624793 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621475    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621477    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621482    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621500    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621503    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621506    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621508    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621512    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621514    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621517    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621520    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621523    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621526    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621529    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621531    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621534    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621537    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621539    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621543    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621547    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.625295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621550    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621552    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621555    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621564    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621567    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621570    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621574    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621577    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621581    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621583    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621586    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621589    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621592    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621594    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621599    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621602    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621604    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621607    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621610    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.625826 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621612    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621616    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621620    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621624    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621626    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621629    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621632    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621635    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621638    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621641    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621643    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621646    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621649    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621652    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621654    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621657    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621660    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621662    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621665    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.626267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621668    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621670    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621673    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621675    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621678    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621681    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621683    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621686    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621690    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621692    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621695    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621698    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621700    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621703    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621706    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621710    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621713    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621716    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621718    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.626743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621722    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621725    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621728    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621730    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.621733    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621818    7037 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621825    7037 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621830    7037 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621835    7037 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621839    7037 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621843    7037 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621847    7037 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621851    7037 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621854    7037 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621857    7037 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621861    7037 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621864    7037 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621867    7037 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621870    7037 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621873    7037 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621876    7037 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621882    7037 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621885    7037 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:00:56.627185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621889    7037 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621892    7037 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621895    7037 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621899    7037 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621902    7037 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621907    7037 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621911    7037 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621914    7037 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621918    7037 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621921    7037 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621924    7037 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621927    7037 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621930    7037 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621933    7037 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621937    7037 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621941    7037 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621944    7037 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621947    7037 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621950    7037 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621953    7037 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621957    7037 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621960    7037 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621963    7037 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621967    7037 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:00:56.628062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621970    7037 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621974    7037 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621977    7037 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621980    7037 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621983    7037 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621987    7037 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621991    7037 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621995    7037 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.621998    7037 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622001    7037 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622004    7037 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622007    7037 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622011    7037 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622015    7037 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622019    7037 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622022    7037 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622025    7037 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622029    7037 flags.go:64] FLAG: --help="false"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622032    7037 flags.go:64] FLAG: --hostname-override="ip-10-0-137-72.ec2.internal"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622035    7037 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622038    7037 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622041    7037 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622045    7037 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:00:56.628777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622049    7037 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622052    7037 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622054    7037 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622057    7037 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622060    7037 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622063    7037 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622066    7037 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622069    7037 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622072    7037 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622076    7037 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622079    7037 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622082    7037 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622084    7037 flags.go:64] FLAG: --lock-file=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622087    7037 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622090    7037 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622094    7037 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622100    7037 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622103    7037 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622106    7037 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622109    7037 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622112    7037 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622116    7037 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622120    7037 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622123    7037 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622127    7037 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:00:56.629327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622131    7037 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622135    7037 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622138    7037 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622141    7037 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622144    7037 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622148    7037 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622151    7037 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622154    7037 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622157    7037 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622164    7037 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622167    7037 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622170    7037 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622173    7037 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622176    7037 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622179    7037 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622182    7037 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622185    7037 flags.go:64] FLAG: --port="10250"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622188    7037 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622192    7037 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea9ad03d3f4cd32a"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622195    7037 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622198    7037 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622201    7037 flags.go:64] FLAG: --register-node="true"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622205    7037 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622209    7037 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622212    7037 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622215    7037 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:00:56.629926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622218    7037 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622222    7037 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622225    7037 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622230    7037 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622233    7037 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622236    7037 flags.go:64] FLAG: --runonce="false"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622239    7037 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622243    7037 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622246    7037 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622249    7037 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622252    7037 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622255    7037 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622258    7037 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622261    7037 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622264    7037 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622268    7037 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622271    7037 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622273    7037 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622277    7037 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622280    7037 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622283    7037 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622288    7037 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622291    7037 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622294    7037 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622298    7037 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:00:56.630640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622301    7037 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622304    7037 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622307    7037 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622311    7037 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622314    7037 flags.go:64] FLAG: --v="2"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622319    7037 flags.go:64] FLAG: --version="false"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622323    7037 flags.go:64] FLAG: --vmodule=""
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622327    7037 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622330    7037 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622455    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622461    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622464    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622467    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622470    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622472    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622475    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622478    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622481    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622484    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622499    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622502    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.631282 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622505    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622508    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622511    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622513    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622516    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622519    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622522    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622525    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622528    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622530    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622533    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622536    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622539    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622542    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622545    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622548    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622551    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622554    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622556    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.631797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622559    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622562    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622567    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622570    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622572    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622575    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622578    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622581    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622584    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622587    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622589    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622592    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622595    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622597    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622600    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622602    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622605    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622608    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622610    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622613    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.632273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622616    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622618    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622621    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622623    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622626    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622631    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622635    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622639    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622643    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622647    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622652    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622656    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622660    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622663    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622668    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622671    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622673    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622677    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622679    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.632927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622682    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622685    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622690    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622694    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622697    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622700    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622703    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622705    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622708    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622710    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622713    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622716    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622718    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622721    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622723    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622726    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622728    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622731    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622734    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622737    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.633620 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622740    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622742    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622745    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622749    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622753    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622756    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622758    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622765    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622768    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622771    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622773    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622776    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622779    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622782    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622784    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.634105 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622787    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.622814    7037 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622848    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622853    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622855    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622859    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622861    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622864    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622867    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622869    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622872    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622875    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622878    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622880    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622883    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.634466 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622886    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622888    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622891    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622894    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622896    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622899    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622901    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622904    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622907    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622911    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622914    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622917    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622920    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622924    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622926    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622929    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622932    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622935    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622938    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622940    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.634896 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622943    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622946    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622948    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622951    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622954    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622956    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622959    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622962    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622964    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622967    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622971    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622973    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622976    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622979    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622981    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622985    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622987    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622990    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622992    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.635479 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622995    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.622998    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623001    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623003    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623006    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623009    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623015    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623019    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623022    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623025    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623028    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623031    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623034    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623037    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623040    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623043    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623045    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623048    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623051    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.636016 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623054    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623057    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623059    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623062    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623065    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623068    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623070    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623073    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623078    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623082    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623085    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623088    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623091    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623094    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623097    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623099    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623102    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623105    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623107    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623111    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.636471 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623114    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623117    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623120    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623122    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623125    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623128    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623136    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623138    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623141    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623144    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623146    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623149    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623152    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623154    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.623157    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.626702    7037 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.626719    7037 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.626738    7037 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.626743    7037 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:00:56.636995 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626799    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626804    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626808    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626811    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626813    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626816    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626819    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626823    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626825    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626828    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626831    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626834    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626837    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626839    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626842    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626845    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626848    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626850    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626853    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626856    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626858    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.637449 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626861    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626865    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626868    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626872    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626874    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626877    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626880    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626883    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626885    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626888    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626890    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626893    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626896    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626898    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626901    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626904    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626907    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626910    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626912    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.637981 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626915    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626918    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626920    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626923    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626926    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626928    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626931    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626934    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626936    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626939    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626942    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626945    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626948    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626950    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626953    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626956    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626959    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626962    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626965    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.638431 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626967    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626970    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626972    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626975    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626979    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626983    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626986    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626989    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626992    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626995    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.626998    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627001    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627003    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627006    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627008    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627011    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627014    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627016    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627019    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.639094 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627021    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627024    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627027    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627029    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627032    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627035    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627038    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627040    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627043    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627046    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627052    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627055    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627058    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627061    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627064    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627067    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627070    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627073    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627076    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.639622 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627080    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627082    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627085    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627088    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627091    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627093    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627096    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627098    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627101    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.627128    7037 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627446    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627826    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627832    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627837    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627842    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.640103 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627850    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627857    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627863    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627867    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627871    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627876    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627886    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627891    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627896    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627907    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627912    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627917    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627921    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627925    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627930    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627935    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627939    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627943    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627948    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.640463 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627957    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627962    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627966    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627970    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627975    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627979    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627984    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627988    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627993    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.627997    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628002    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628006    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628015    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628019    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628024    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628028    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628033    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628037    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628042    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628047    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.640935 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628051    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628056    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628061    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628066    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628071    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628081    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628086    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628091    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628096    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628100    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628105    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628112    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628117    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628122    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628126    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628131    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628135    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628145    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628150    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.641735 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628155    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628159    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628164    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628168    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628173    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628178    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628183    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628187    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628191    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628195    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628200    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628209    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628213    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628218    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628222    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628226    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628230    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628236    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628239    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628242    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.642239 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628245    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628247    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628251    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628253    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628256    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628259    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628262    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628265    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628268    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628271    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628277    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628280    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628283    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628286    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628289    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628292    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628295    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628298    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628301    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628303    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.642754 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628306    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628311    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628315    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.628370    7037 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628479    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628504    7037 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628508    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628612    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628616    7037 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628619    7037 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628622    7037 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628626    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628629    7037 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628632    7037 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628635    7037 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628638    7037 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:56.643273 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628642    7037 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628644    7037 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628647    7037 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628650    7037 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628653    7037 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628656    7037 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628659    7037 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628662    7037 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628664    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628667    7037 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628670    7037 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628673    7037 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628676    7037 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628678    7037 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628681    7037 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628683    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628687    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628690    7037 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628693    7037 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:56.643675 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628696    7037 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628698    7037 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628704    7037 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628709    7037 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628712    7037 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628715    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628718    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628721    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628724    7037 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628727    7037 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628730    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628733    7037 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628736    7037 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628739    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628742    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628745    7037 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628747    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628750    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628753    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:56.644137 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628757    7037 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628760    7037 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628763    7037 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628766    7037 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628768    7037 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628771    7037 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628774    7037 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628777    7037 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628780    7037 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628783    7037 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628789    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628792    7037 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628795    7037 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628798    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628801    7037 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628805    7037 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628807    7037 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628810    7037 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628813    7037 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628818    7037 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:56.644646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628821    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628824    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628826    7037 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628829    7037 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628832    7037 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628835    7037 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628837    7037 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628840    7037 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628843    7037 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628846    7037 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628849    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628851    7037 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628854    7037 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628858    7037 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628863    7037 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628866    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628870    7037 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628873    7037 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628875    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:56.645133 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628879    7037 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628881    7037 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628886    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628890    7037 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628893    7037 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628896    7037 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628899    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628901    7037 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628904    7037 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628906    7037 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628909    7037 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628912    7037 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628915    7037 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628917    7037 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628920    7037 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628923    7037 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:00:56.628925    7037 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.628980    7037 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.629940    7037 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:00:56.645604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.629994    7037 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.630170    7037 server.go:1014] "Starting client certificate rotation"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.630319    7037 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.630378    7037 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:54:44 +0000 UTC" deadline="2028-04-10 08:33:08.487436032 +0000 UTC"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.630420    7037 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12659h32m11.857022462s"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.630648    7037 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.631664    7037 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.632764    7037 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.634780    7037 log.go:25] "Validated CRI v1 image API"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.635520    7037 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:00:56.646380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.637862    7037 fs.go:135] Filesystem UUIDs: map[2f2bb360-add7-46f6-aa12-10d78fc81670:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cb38f94a-a4c0-4cd8-b3c5-7865e325bdd7:/dev/nvme0n1p4]
Oct 30 21:00:56.646944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.637882    7037 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829/userdata/shm major:0 minor:483 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/381d873ef40bf89608a47b7892c820ef3c65254b998e32b886c5f453788d4445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/381d873ef40bf89608a47b7892c820ef3c65254b998e32b886c5f453788d4445/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f5fc37f005cd26e28357b4f0e46e6f64b67c18191c1d762048948f1316d9f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f5fc37f005cd26e28357b4f0e46e6f64b67c18191c1d762048948f1316d9f2b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b/userdata/shm major:0 minor:393 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bbf8b47-e869-43e4-8fdd-459639f068c3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0bbf8b47-e869-43e4-8fdd-459639f068c3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~projected/kube-api-access-74ntk:{mountpoint:/var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~projected/kube-api-access-74ntk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~projected/kube-api-access-q5bz8:{mountpoint:/var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~projected/kube-api-access-q5bz8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~secret/metrics-certs major:0 minor:486 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b36017a-ec01-4979-a358-7198261684d7/volumes/kubernetes.io~projected/kube-api-access-97x6l:{mountpoint:/var/lib/kubelet/pods/2b36017a-ec01-4979-a358-7198261684d7/volumes/kubernetes.io~projected/kube-api-access-97x6l major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/alertmanager/9 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~empty-dir/config-out major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/kube-api-access-hlsx4:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/kube-api-access-hlsx4 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/tls-assets major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/config-volume major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/web-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/config-reloader/2 major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/prometheus/15 major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/kube-api-access-49dqm:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/kube-api-access-49dqm major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/tls-assets major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/config major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/web-config major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~projected/kube-api-access-vgh27:{mountpoint:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~projected/kube-api-access-vgh27 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~projected/kube-api-access-kqk8r:{mountpoint:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~projected/kube-api-access-kqk8r major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7871fe29-fbbf-4733-878f-680fb56cee31/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7871fe29-fbbf-4733-878f-680fb56cee31/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~projected/kube-api-access-wm67s:{mountpoint:/var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~projected/kube-api-access-wm67s major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~projected/kube-api-access-s5f8d:{mountpoint:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~projected/kube-api-access-s5f8d major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~projected/kube-api-access-8nm74:{mountpoint:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~projected/kube-api-access-8nm74 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e300873-ed21-450b-9e6b-810ee569b783/volumes/kubernetes.io~projected/kube-api-access-6ggpw:{mountpoint:/var/lib/kubelet/pods/8e300873-ed21-450b-9e6b-810ee569b783/volumes/kubernetes.io~projected/kube-api-access-6ggpw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~projected/kube-api-access-k2bkz:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~projected/kube-api-access-k2bkz major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4/volumes/kubernetes.io~projected/kube-api-access-qmzhk:{mountpoint:/var/lib/kubelet/pods/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4/volumes/kubernetes.io~projected/kube-api-access-qmzhk major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~projected/kube-api-access-lhxd4:{mountpoint:/var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~projected/kube-api-access-lhxd4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~projected/kube-api-access-2pnzh:{mountpoint:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~projected/kube-api-access-2pnzh major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~projected/kube-api-access-2vh7w:{mountpoint:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~projected/kube-api-access-2vh7w major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593/volumes/kubernetes.io~projected/kube-api-access-thm44:{mountpoint:/var/lib/kubelet/pods/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593/volumes/kubernetes.io~projected/kube-api-access-thm44 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~projected/kube-api-access-d2dp2:{mountpoint:/var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~projected/kube-api-access-d2dp2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c56952e8-b525-401d-8128-e4c8ac0d6469/volumes/kubernetes.io~projected/kube-api-access-jm2sb:{mountpoint:/var/lib/kubelet/pods/c56952e8-b525-401d-8128-e4c8ac0d6469/volumes/kubernetes.io~projected/kube-api-access-jm2sb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~projected/kube-api-access-pgxfr:{mountpoint:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~projected/kube-api-access-pgxfr major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~projected/kube-api-access-57bjx:{mountpoint:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~projected/kube-api-access-57bjx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e76c52e5-2903-48df-92ea-bd07b934c773/volumes/kubernetes.io~projected/kube-api-access-ckkd4:{mountpoint:/var/lib/kubelet/pods/e76c52e5-2903-48df-92ea-bd07b934c773/volumes/kubernetes.io~projected/kube-api-access-ckkd4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~projected/kube-api-access-g5v2k:{mountpoint:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~projected/kube-api-access-g5v2k major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f/volumes/kubernetes.io~projected/kube-api-access-l2zb2:{mountpoint:/var/lib/kubelet/pods/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f/volumes/kubernetes.io~projected/kube-api-access-l2zb2 major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1f70e2d3b898aca016817f5fdb5c3c09bdc519ceb9556d4d8621ed48409f9956/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4446ffe73e00a5060bb8f8046b41378619204b330452ce06e020f9764361d997/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8088f7347880aef34096e96957d67275bf9928e2ad03e5ec3816826320c472f7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e76c350e1b2796e148fcbeaadbabdefbf71ee5c7fb8f2f431a0a18ffc3bca597/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e28b0a47f15ebdb2b0ab22eaa40945a987b1227cd2a6ceba6c0160a97b69a4a3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/82da48d81a0b1438a9f665adba142dca426974335a4853ff3baa322231fa5540/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7f59bec740b54cf7dfe3fbd9606e2878ebf45a64e3322d76b261be5065926790/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8372f73e5c900e64cbe2d24ecca4e6e0440846390f96f3ddc1ade17952ffda3b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/817c5c07565dc5f5dbe27e98861a4b3161d290bc86bc206aab173d70e74e3ccb/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/71d231fc5806d3cd15961401c9e99a51e7214614535363bfc14fc56cd6bd548b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/d8c8836a9921ad60bda73a1034bc68017b2a3f40204fb19a6f060b0fe762e91b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/3935fbcfa52376866ef81dedd896488f85f5e4ba7d605fae0b3b2fa0343d4b28/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4e8cd80b8f941b8f35566824283049b845b4d0474558dbb08e10e2c880164723/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/998354938ee326ce19497a6f612c96dc3767bc5a3aa2a873d9223a381c4e3768/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bbca2c2b63339eb54590ed2c03f4d21ee9f8bb7a114e45d4cfcadbe96a120041/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/34a2eb315000f45836ac73309505c3ca8ae39f7ccb9f7e51df9d2aef0d59fa78/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6b34d1a4c80d2b4aad446ce30e9740c6c6a96a42003e2b557377efe013e2e66e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/92d278fbdd5a9b5869077a1f4dafc394b313d67da6c1dcf145828c01c76af471/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5fba38b81ad4cbff4c0d61d25d5b31371af84454dc1be438e0c686a1b0d40559/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6cfad9247b3aec95fca199e3ee19c2da5afb91b61b599a4f032b0c6db2a971d9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c94e66285282cd6b21560340e8cb0b59cdd1d10459df9a98ab5b8e942d4bb5e1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/295858a2d71f5c434d949920d91349224d6c91a55a65509c948eb781886c4b0d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6ba8f5df00b0dbe9d506a0a00fdfe5a8cfc0964509a71574e205008d9fcad536/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/51580fd90c9e2cc33cc32cffe92b2ebb8e0173ef3a4b0b5b241cda88ecb1e56a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/6982759b3a68393152bca7789975d4c722d5be1bfce0b5d5570ed21296985d96/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3a56df9a7fcdd163573c9bf753c95eef8d434ae9bf13a6ec58a695490b56d586/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/ee9005e3e5273dce689f16e297580471db583479c29d142b6a654285a8c50c7a/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d10cb5e0771027a1fde9e6f9ec93e2b9e94746d8d6d5f81552138505e8e37f03/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/1b9d6517aa37ca4881c09ff68a9f0b33300065f8f47bcbb1fa7d10f010ee6efd/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d501ff37d756b4a1d8723bf8db402dfcf249b97a1b7ea655b1677bb2e65d48f7/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/cc26d096fdc3b340e0994fd42e7f23704e82f1581e5cec9b7d3854f698e9f9cc/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2f416216b8da4edd2a365a4250710682f05b894ca940c99f2b61e82d2b2b5dc3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ec08d219e964914952b99f0b12026ba596a11c11680b5e532504b81c6fbfcb53/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/330ec565fa8118e5f724cb7cb29371d7664e67d043bdabad2164e84ff16a9a27/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/2b9a6fbaf683d511cdedbb7ace02f6dbebc22edf05f4a694483f26299a67f1d9/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/814a57f4d59bb9fc5f5d9858c0c6c841c0f7886b71ca736f14ed0f357225de58/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/94892697be570f88b303630766343f13d857fd5005e660e221bf2dd5029658dd/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/26a82484a18af0531b2654b2e501e68c390730e35ec056175ff1059fad91d300/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d6d768791082776fc1517c3518e4d6b437f8281f94275f2ada694ee8b3a3d969/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/34e7958db67766376f1e66b06160219694693b5826975f1e81e5cf49416ab378/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/c2704ffd6e2f581729361b161fc77f36e30f7d73198a2034b0b34901e62fae11/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/0590cc921204a72b93c513788d941bef7ce9349d4d32c132398e96388a4c625c/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/2a4a732ee82e758766a780cea04a099c7b6984db1a050e2c7ed9750e51738955/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/534873a7da0b6256b2c758a8e4a2a05bdf921ab37756633686c5b1683ad057d4/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/7561a002830b0838e145c70c69fc6923e18d5f97f0dff2df97f19c2d43f19013/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/cda63d0fafe4b1c6939f541d68ac6cce22089fb2c93370f1073efc7e4ae7c0c4/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/a43a82903555022450790b12d7892361cbd6189c200ebd9c82104c3c08e06df3/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/8f0b93db05f869b2b39102ea1e0331297400cba040ed557aad055294ee37ca59/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/b5a36fbdaf78f5b59b572a428d3c30666bf8f8dd99bae1519d54fe3fdfbc8fbf/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/d88dc27a6b26253052464ca30722c214a6fcccf542463ab4d1dc272068d3ebc0/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/bfbb61adc781a02f272ac5f7e5e491da3441c7f46fbf68ac733d0439e806e201/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/615d18d7b0f273c81aaf737e3035259d96f50af723404bf740e756c5f0029945/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/6e2aeb9c2ccf4447fbf104c1f50f415be49f545b23ad77dfa8ee8089bf88451b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/50a1b14c4623a4e6c96b21f8aaec11cd35b2992dc2975ea0e1d113ac8ba48a0e/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/3dcb291a6e08515e28aa7561c8a2bb18749e53dd740bb57dbecc7d846e44148a/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/c85c5e22f7e6b7fdd186cc11d5b871bad3f17de93e1a8724160400aa31ef1841/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/3cd0d4b7a2879b908428b613b04526e03db9a263c5071bf5774d10627d93afd4/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/7c372bce147c39435aea329c0f8bbccc3192c7fdbd41faceacf3a678f39e6465/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/b7e8bac264883b459d4eddf488dbcc8c0e76a615de96349a4a917cccc7faa37b/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/9cbc80c4a4401e4eac9fa9cb62e44df9d35f2f473738ddbb3d15272b45ec2edb/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/2a7e196567fb7dfea04c083ef524e449cbaf27251066c4e6dfe25635da66c06b/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/250a519beda7e02c86a0f8c0117dbd1688cee7469960090a7ffa91477f9f31c1/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/3afe5297e36c00de4f456d8a89778e66edbe0494578410de989edda59bd6e9ad/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/7c2ef19dde3b0eafb534cbf9eda333c2ce99c682385251513d5deb95810f3921/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/808d2b3407bee84198ace60375f548885ad1354afbce5e471a7df3c38fe32f1e/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/37eb703d0aa7941867d49d758b7e723f665871923dc32844c6d7edcf046d0887/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/786b1b358674f6e4f190acc193d4ac00b3b1818382a97b2bc9a9e35c83855d99/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/84d22b7b1cfbaaca4e524cbdb351e4275a8b6538cefadded1f926ab2a5e82b94/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/8cecdce44e87131cb20bfd20b40f1a450021414e4a4285ecddf8fb162b78c8fe/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/0cd32e4cb9a718ba137341e8e4d6cdc6bd65c84296bdabbf752ddeff3216cdf7/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/6415836d968b9e866555a88e811e45e9b11e85e6cf0acd018c3c74b24e04ae39/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/27bf91c2b369b492a69c4476fcb7eced6c9a5615215a1403ce1bd5955a89cdb4/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/afd6366b5dec0c648d5f2ab88fece56c8a06c372a3abce3e9f1aef01f19b9fae/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/e823675f3c07056d33581d6c250369028eff3cb21242a47c7a9a3156a957efb5/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/ef1d4ccf0254e134a3daa9809a24b187c1f6cb1f854cdbb9c37b5deb072e0546/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/d0ab50017f6e4e88e2570eb77185c770320f1908a78e827755989b86286dbdbf/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/e80963354e4d19163c41652c02b7059a188cd9d9d3b559f8ec2f47f64437f4fc/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/535796c82db3e7411417a450f153d8f6af0c998a3c5094c1c74a32347e599b1a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-530:{mountpoint:/var/lib/containers/storage/overlay/ef33bb954cde4f85d5a8daa38e3b262b1943ae339244751be7ceefb0dcd787f6/merged major:0 minor:530 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/56f491821eeda7e2f8771336882c96e1c7b2fcbc079af5bf85770e9c06b802f3/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/e69d9bcf0e7f5efe700bd91476aeb5fa6166cedce9b6ee057424c9ab0974073f/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-545:{mountpoint:/var/lib/containers/storage/overlay/8854630440f903b03967e610f7d45706b02f3a52b7446068b99b6393be246635/merged major:0 minor:545 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/b074fc40f3ac0fbab71308c44baa41731677414e4ca51e7456ffe7e409f3ab4f/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/58f4b5448a38e6136327739cfa97550f5d178c2bdc01a95e68895e163b70f14f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7c21e6b5544f6a67975283efdad79e634359e4ec7f2db0c261f76ccad23ccb4f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fe1afc9d15df1c8e72a50129cc7ab46e26df1d743ecddb3a5388619cd900afce/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/81b4062ef0415070e5930ca6a6833c29728b02432aa9e7aca3d041094d87f274/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/55851c1e5e85b95275da4363e5c1b7242c91eed513a30ebaa30f47adafe918e9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0a4231ffa763dc4fa6b753222ec75b60cee8b109a107ab1fb2c760d642360a30/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f4a8b372cc13c4d2d8b640691e4f9ce466877c1dab3e1b6b3c5d4e05efdacdf9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:00:56.648109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.647611    7037 manager.go:217] Machine: {Timestamp:2026-10-30 21:00:56.646783977 +0000 UTC m=+0.058323404 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105301 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6efa917121a345c1956545e75325 SystemUUID:ec2c6efa-9171-21a3-45c1-956545e75325 BootID:802a7bb7-9f09-4f6f-9a0f-61107a36ca64 Filesystems:[{Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~projected/kube-api-access-74ntk DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~projected/kube-api-access-g5v2k DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-545 DeviceMajor:0 DeviceMinor:545 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/839d358db6db76cac1d7da55dca93fc55702473b9b9535f9bfa23aa9c09862d3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:471 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~projected/kube-api-access-q5bz8 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4/volumes/kubernetes.io~projected/kube-api-access-qmzhk DeviceMajor:0 DeviceMinor:488 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~projected/kube-api-access-57bjx DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ed64a4b034cf4b2dbfb30a065b51eb7e4021192fb3bbd17c3b48f6aa07463fd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:472 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aa06446db38ccbd00ce413e170e85843e03e728654ed6614f66b648cb08c76b/userdata/shm DeviceMajor:0 DeviceMinor:393 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~projected/kube-api-access-lhxd4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c56952e8-b525-401d-8128-e4c8ac0d6469/volumes/kubernetes.io~projected/kube-api-access-jm2sb DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f/volumes/kubernetes.io~projected/kube-api-access-l2zb2 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e76c52e5-2903-48df-92ea-bd07b934c773/volumes/kubernetes.io~projected/kube-api-access-ckkd4 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/381d873ef40bf89608a47b7892c820ef3c65254b998e32b886c5f453788d4445/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~projected/kube-api-access-s5f8d DeviceMajor:0 DeviceMinor:399 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e300873-ed21-450b-9e6b-810ee569b783/volumes/kubernetes.io~projected/kube-api-access-6ggpw DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/kube-api-access-hlsx4 DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/kube-api-access-49dqm DeviceMajor:0 DeviceMinor:475 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f2e0de919992040a2d76fb9fb482cc5838a4540b2927b7a6594b62aa271dd9b/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da5ac24fe9cfcd2385e38a4dbb7c8ebcc05607a94e40e5efe579116396c6f417/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~projected/kube-api-access-kqk8r DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:388 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593/volumes/kubernetes.io~projected/kube-api-access-thm44 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:460 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~projected/kube-api-access-pgxfr DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bef5e49abc6fbce4af2d2f4f7f2b69e391dc52e2d6caebf7882e7af7c749219b/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:397 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:364 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b7a705f6b24d0e717cf2a8d0c4cdfea918037ba20dc92bb49206b01ae7a34da/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6a1e9f7b78a2a7fb717985323e611b5b9e380273fa9e0d75d7863f1c560438c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baae11a916340577b6caa76c0930e5bf2b3c7779b38608304c88053ef20817c4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f5fc37f005cd26e28357b4f0e46e6f64b67c18191c1d762048948f1316d9f2b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:398 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7871fe29-fbbf-4733-878f-680fb56cee31/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-530 DeviceMajor:0 DeviceMinor:530 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b088453c-eec1-465f-939e-e4c78cc2290b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59681ff5a228b8e93ac70eded60dd49fd61fcce4b87f99abcf713f85d309cedb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b745be82e80ea9565dd9834c5490511aaff32b4052aa8d7e03d4bae006e394c/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:467 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f1ff389-d912-4771-8a8f-43b6ab842829/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:486 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~projected/kube-api-access-vgh27 DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:468 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:470 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829/userdata/shm DeviceMajor:0 DeviceMinor:483 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:474 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20e839b568eda447d1f18cbe4e202a8e719ade727ab5185a2caf507318953e81/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/548cadb6-b5c0-4f7d-ad73-e5060de08fc9/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:473 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31e1062e7772b42df1e0219553b6f7c04c1f7fd04d146a95cd0b1b3e3ffbdebb/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:465 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1927898bc51bc2d2cf120d659f448bf71b0f0a843d9a624838805ff74240c6ef/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c5ef48d340302b3f38bf1b0f80fe1149058a27589f7289a946184fcacb8bd29/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes/kubernetes.io~projected/kube-api-access-2pnzh DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3/volumes/kubernetes.io~projected/kube-api-access-2vh7w DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~projected/kube-api-access-k2bkz DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:466 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8062ab8-4d4e-41a1-9ed0-1328ffc23870/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1a737db3fe9c8c7689dc3d659015336cdc0a816c5fb1eb3eaef66d01a46dc7e/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1137ad4d6a89fd616f9a82d1ce9522de572208bc80d4618b2e578b0e791027a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b36017a-ec01-4979-a358-7198261684d7/volumes/kubernetes.io~projected/kube-api-access-97x6l DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~projected/kube-api-access-d2dp2 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/943e6093-44ff-4e3e-97d4-67a24ce560a0/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/volumes/kubernetes.io~projected/kube-api-access-wm67s DeviceMajor:0 DeviceMinor:392 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ea0220-eb35-4f3d-8147-060179390f3d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bbf8b47-e869-43e4-8fdd-459639f068c3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:487 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87e040ec-3520-40f4-b906-cc22de4d532a/volumes/kubernetes.io~projected/kube-api-access-8nm74 DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/934d37e1d067403811b54fbbd11b0c451a557b521580b961f08fe29ff44f104c/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce43bf79-f468-4826-ae78-e58f103f9fa7/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:469 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd1b8b2546ad57b5e9d08fb7039a7fb41e5ea0a4d543b50554e3aa3bc833369c/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c5ef48d340302b MacAddress:76:ef:4d:3a:9d:33 Speed:10000 Mtu:8901} {Name:1927898bc51bc2d MacAddress:2e:73:e5:9b:b9:71 Speed:10000 Mtu:8901} {Name:1b7a705f6b24d0e MacAddress:76:46:a9:d5:7d:0f Speed:10000 Mtu:8901} {Name:1db17f887334328 MacAddress:ce:9a:33:07:06:b2 Speed:10000 Mtu:8901} {Name:2f6488c76e4332d MacAddress:f6:f1:fd:3b:8d:11 Speed:10000 Mtu:8901} {Name:38c26875765559a MacAddress:f2:0e:26:a7:71:83 Speed:10000 Mtu:8901} {Name:59681ff5a228b8e MacAddress:0a:41:21:f3:47:9f Speed:10000 Mtu:8901} {Name:6f2e0de91999204 MacAddress:9a:75:55:26:fd:80 Speed:10000 Mtu:8901} {Name:7f5fc37f005cd26 MacAddress:9e:c3:ff:d7:85:99 Speed:10000 Mtu:8901} {Name:8127bfb6ceb6ad1 MacAddress:0e:28:10:5a:50:60 Speed:10000 Mtu:8901} {Name:839d358db6db76c MacAddress:82:c2:b2:b8:8e:c9 Speed:10000 Mtu:8901} {Name:8b745be82e80ea9 MacAddress:e6:02:2b:79:78:92 Speed:10000 Mtu:8901} {Name:934d37e1d067403 MacAddress:46:9c:45:44:ea:0b Speed:10000 Mtu:8901} {Name:9aa06446db38ccb MacAddress:7e:22:c5:2b:8c:35 Speed:10000 Mtu:8901} {Name:bef5e49abc6fbce MacAddress:c2:22:c0:0f:1f:82 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:25:18:3d Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:a0:fd:8e:92:c7 Speed:0 Mtu:8901} {Name:dd1b8b2546ad57b MacAddress:3e:bf:b5:c9:e6:d7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:25:18:3d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:29:0a:2c:3d:a4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:ca:8b:0e:42:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:00:56.648388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648115    7037 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:00:56.648388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648160    7037 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:00:56.648483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648455    7037 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:00:56.648620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648485    7037 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:00:56.648663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648623    7037 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:00:56.648663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648631    7037 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:00:56.648663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648645    7037 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:56.648741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648665    7037 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:56.648769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648757    7037 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:00:56.648915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.648905    7037 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:00:56.649030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649021    7037 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:00:56.649081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649072    7037 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:00:56.649111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649087    7037 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:00:56.649111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649098    7037 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:00:56.649111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649107    7037 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:00:56.649240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649115    7037 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:00:56.649902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649886    7037 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:00:56.649902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.649899    7037 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:56.650420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.650402    7037 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:00:56.653103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653088    7037 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:00:56.653103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653102    7037 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:56.653213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653190    7037 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:00:56.653370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653359    7037 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:00:56.653406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653387    7037 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:00:56.653478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653470    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653485    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653507    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653515    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653521    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653528    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:00:56.653533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653533    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653539    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653548    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653554    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653577    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653586    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653615    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:00:56.653710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.653620    7037 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:00:56.656928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.656916    7037 server.go:1263] "Started kubelet"
Oct 30 21:00:56.657093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.657023    7037 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:00:56.657144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.657121    7037 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:00:56.657609 ip-10-0-137-72 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:00:56.658043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.657627    7037 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:00:56.658043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.657684    7037 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:00:56.660726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.660708    7037 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:00:56.665945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.665926    7037 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:00:56.666041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.665957    7037 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:00:56.666041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.665991    7037 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:00:56.666041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.666007    7037 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:00:56.666041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.666018    7037 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:46 +0000 UTC" deadline="2028-04-17 18:07:30.032136285 +0000 UTC"
Oct 30 21:00:56.666041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.666039    7037 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12837h6m33.366100266s"
Oct 30 21:00:56.666281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.666105    7037 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:00:56.666281 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:56.666128    7037 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-72.ec2.internal\" not found"
Oct 30 21:00:56.667519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.667461    7037 factory.go:55] Registering systemd factory
Oct 30 21:00:56.667519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.667484    7037 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:00:56.668005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.667983    7037 factory.go:153] Registering CRI-O factory
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.668007    7037 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.668081    7037 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.668103    7037 factory.go:103] Registering Raw factory
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.668120    7037 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.668441    7037 manager.go:319] Starting recovery of all containers
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669150    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669190    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669206    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669220    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b36017a-ec01-4979-a358-7198261684d7" volumeName="kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669232    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669245    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669257    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e300873-ed21-450b-9e6b-810ee569b783" volumeName="kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669270    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669287    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e040ec-3520-40f4-b906-cc22de4d532a" volumeName="kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669300    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669312    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669323    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669336    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669348    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669365    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669376    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76c52e5-2903-48df-92ea-bd07b934c773" volumeName="kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669387    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8062ab8-4d4e-41a1-9ed0-1328ffc23870" volumeName="kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669402    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669413    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669425    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4" volumeName="kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669440    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ea0220-eb35-4f3d-8147-060179390f3d" volumeName="kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669452    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669466    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669478    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c56952e8-b525-401d-8128-e4c8ac0d6469" volumeName="kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669506    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669520    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbf8b47-e869-43e4-8fdd-459639f068c3" volumeName="kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669532    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669543    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669571    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669610    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669625    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669637    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c" volumeName="kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669650    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c" volumeName="kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669663    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c" volumeName="kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669675    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669727    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669747    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669760    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c56952e8-b525-401d-8128-e4c8ac0d6469" volumeName="kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669774    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f1ff389-d912-4771-8a8f-43b6ab842829" volumeName="kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669787    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669801    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669813    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669849    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669861    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8062ab8-4d4e-41a1-9ed0-1328ffc23870" volumeName="kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669876    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669888    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ea0220-eb35-4f3d-8147-060179390f3d" volumeName="kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669901    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f" volumeName="kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669915    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669926    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669941    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db" volumeName="kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669956    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76c52e5-2903-48df-92ea-bd07b934c773" volumeName="kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669969    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669983    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.669996    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670008    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670023    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e040ec-3520-40f4-b906-cc22de4d532a" volumeName="kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670065    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670078    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670092    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670107    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f" volumeName="kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670125    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b36017a-ec01-4979-a358-7198261684d7" volumeName="kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670139    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670158    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ac9b7b-1e1f-4966-8522-d92af9ee73b3" volumeName="kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670174    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670191    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670204    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670220    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8211ee52-aa1e-4a0c-bfe8-3e1779da22a7" volumeName="kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670235    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670260    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh" seLinuxMountContext=""
Oct 30 21:00:56.670204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670276    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670289    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670320    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670335    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670347    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ea0220-eb35-4f3d-8147-060179390f3d" volumeName="kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670361    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670372    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f1ff389-d912-4771-8a8f-43b6ab842829" volumeName="kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670384    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670396    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670409    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670430    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ea0220-eb35-4f3d-8147-060179390f3d" volumeName="kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670442    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7871fe29-fbbf-4733-878f-680fb56cee31" volumeName="kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670454    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e300873-ed21-450b-9e6b-810ee569b783" volumeName="kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670467    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8062ab8-4d4e-41a1-9ed0-1328ffc23870" volumeName="kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670504    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670554    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670566    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7871fe29-fbbf-4733-878f-680fb56cee31" volumeName="kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670579    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8211ee52-aa1e-4a0c-bfe8-3e1779da22a7" volumeName="kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670591    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8062ab8-4d4e-41a1-9ed0-1328ffc23870" volumeName="kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670603    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670616    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670627    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670639    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670655    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce43bf79-f468-4826-ae78-e58f103f9fa7" volumeName="kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670669    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db" volumeName="kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670685    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670699    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670710    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670722    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db" volumeName="kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670797    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670857    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593" volumeName="kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670871    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f" volumeName="kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670910    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e300873-ed21-450b-9e6b-810ee569b783" volumeName="kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670922    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b088453c-eec1-465f-939e-e4c78cc2290b" volumeName="kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670936    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670948    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670961    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548cadb6-b5c0-4f7d-ad73-e5060de08fc9" volumeName="kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670974    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e040ec-3520-40f4-b906-cc22de4d532a" volumeName="kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.670988    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b088453c-eec1-465f-939e-e4c78cc2290b" volumeName="kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671004    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671016    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ac9b7b-1e1f-4966-8522-d92af9ee73b3" volumeName="kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671031    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ac9b7b-1e1f-4966-8522-d92af9ee73b3" volumeName="kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671044    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671088    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671101    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" volumeName="kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671111    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="826b8c4d-26f6-47f8-a885-85c93b1996cf" volumeName="kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671134    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e300873-ed21-450b-9e6b-810ee569b783" volumeName="kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671149    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db" volumeName="kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671161    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b802282-8c04-4522-8d63-1d3b66d897aa" volumeName="kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671173    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671185    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453a3396-2bff-41c7-8164-25a8991c4e2d" volumeName="kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671196    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b23412f3-007f-4358-a2a1-09288ff60856" volumeName="kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671208    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c" volumeName="kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671223    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87e040ec-3520-40f4-b906-cc22de4d532a" volumeName="kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671235    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="943e6093-44ff-4e3e-97d4-67a24ce560a0" volumeName="kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671247    7037 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db" volumeName="kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671259    7037 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:00:56.673524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.671270    7037 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:00:56.697712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.697659    7037 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:00:56.699179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.699154    7037 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:00:56.699179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.699180    7037 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:00:56.699337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.699200    7037 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:00:56.699337 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:56.699242    7037 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:00:56.710723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.710697    7037 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:56.710853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.710807    7037 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:00:56.711520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.711482    7037 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:56.715878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.715847    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="d0159d09e37f3f934914c9cd69bce4ae05309ea35e90c490c3329541151d95f3" exitCode=0
Oct 30 21:00:56.724813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724789    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="e783eee8eb6fead7ab6db0e1267a635802dfa79afe5bfbec781f23569c1ec660" exitCode=0
Oct 30 21:00:56.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724815    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="934e17bc9087d81c06c6c3481f474f8507ed5cca1e0ae4e12b4860489ac78f38" exitCode=0
Oct 30 21:00:56.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724827    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="6048b16741b376a01ff81d2d5614350893e6c775b6ea6ebb8dc7c27310064f8e" exitCode=0
Oct 30 21:00:56.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724835    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="45bd75c49893f17cc3f4ef48b36047b89e43d914de653ef951ec30ea5b5f0f70" exitCode=0
Oct 30 21:00:56.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724842    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="8429dd771c43723257c0d19f5e6cfebf6536b721a585b60fb9fcfc781e380b5f" exitCode=0
Oct 30 21:00:56.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.724848    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e300873-ed21-450b-9e6b-810ee569b783" containerID="8e6e62b073e941864ed45b85baaac6a2a3838c2aa7f4002225edaa1d1206aebe" exitCode=0
Oct 30 21:00:56.735975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.735952    7037 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:56.736555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.736535    7037 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:56.740766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.740743    7037 generic.go:356] "Generic (PLEG): container finished" podID="be11cf10dd8daf22a5ab8288141ce668" containerID="52d3bbc0c5bcb0c50f2efc6d172b86ec152c3b92777ff246161dc37825313405" exitCode=0
Oct 30 21:00:56.756599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.756571    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699" exitCode=0
Oct 30 21:00:56.758757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.758731    7037 generic.go:356] "Generic (PLEG): container finished" podID="6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19" containerID="2951873accf1d2eb91d98be2a7c994f095fb00a3a565235b5c624a7eac06bb60" exitCode=0
Oct 30 21:00:56.799385 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:56.799284    7037 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:00:56.800824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.800806    7037 manager.go:324] Recovery completed
Oct 30 21:00:56.826985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.826959    7037 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:00:56.826985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.826977    7037 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:00:56.827105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.826998    7037 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:00:56.827136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827118    7037 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:00:56.827167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827130    7037 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:00:56.827167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827146    7037 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:00:56.827167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827151    7037 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:00:56.827167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827156    7037 policy_none.go:50] "Start"
Oct 30 21:00:56.827167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827165    7037 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:00:56.827325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.827173    7037 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:00:56.828266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.828253    7037 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:00:56.828300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.828267    7037 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:00:56.828300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.828273    7037 policy_none.go:44] "Start"
Oct 30 21:00:56.833227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.833201    7037 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:00:56.835945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.835932    7037 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:00:56.836885 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:56.836825    7037 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:00:56.836965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.836888    7037 server.go:87] "Starting device plugin registration server"
Oct 30 21:00:56.837220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837200    7037 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:00:56.837331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837230    7037 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:00:56.837473 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837452    7037 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:00:56.837631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837596    7037 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:00:56.837794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837642    7037 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:00:56.837794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837650    7037 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:00:56.837881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.837733    7037 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:00:56.837630222Z","UUID":"a954c103-c2ab-4584-8525-9b234a411db8","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:00:56.840589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.840568    7037 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:00:56.840679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.840608    7037 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:00:56.840788 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:56.840765    7037 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:00:56.937904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.937877    7037 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:56.939871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.939855    7037 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:56.939943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.939883    7037 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:56.939943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.939893    7037 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-72.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:00:56.940011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.939967    7037 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-72.ec2.internal"
Oct 30 21:00:56.974572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.974552    7037 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-72.ec2.internal"
Oct 30 21:00:56.974650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.974606    7037 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-72.ec2.internal"
Oct 30 21:00:56.999414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:56.999370    7037 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"]
Oct 30 21:00:57.000535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000511    7037 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.000911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000858    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal" event={"ID":"ca4d5a6e1cac786a378b125c5389b555","Type":"ContainerStarted","Data":"2e65360950122a7e50dc09737efdfb5c196ca34b1ac9e5d27c627208e52f56e9"}
Oct 30 21:00:57.000975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000927    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal" event={"ID":"ca4d5a6e1cac786a378b125c5389b555","Type":"ContainerStarted","Data":"ce25cdc8d9f196d5ab20251aa08a67c7cdada31acf20c5ababa8e5f7e06ec78e"}
Oct 30 21:00:57.000975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000954    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerStarted","Data":"981bbf3fa74d9811e2a13115e142c9531239cf3f29349e74da13bd27be96f72a"}
Oct 30 21:00:57.001069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000974    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerDied","Data":"52d3bbc0c5bcb0c50f2efc6d172b86ec152c3b92777ff246161dc37825313405"}
Oct 30 21:00:57.001069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.000998    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal" event={"ID":"be11cf10dd8daf22a5ab8288141ce668","Type":"ContainerStarted","Data":"22b9a661709a5db0e64214df995376bbcef19fb2a1f542ef81e97dd96f775bcf"}
Oct 30 21:00:57.001154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.001077    7037 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.025750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.025725    7037 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:57.025837 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:57.025772    7037 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.026199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.026186    7037 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:57.026247 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:57.026219    7037 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.112583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.112554    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.112720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.112584    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.112720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.112604    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213093    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213126    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213193    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213200    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213211    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be11cf10dd8daf22a5ab8288141ce668-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" (UID: \"be11cf10dd8daf22a5ab8288141ce668\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.213271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.213235    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca4d5a6e1cac786a378b125c5389b555-config\") pod \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" (UID: \"ca4d5a6e1cac786a378b125c5389b555\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.649759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.649726    7037 apiserver.go:52] "Watching apiserver"
Oct 30 21:00:57.705006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.704970    7037 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:00:57.705729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.705695    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal","openshift-insights/insights-runtime-extractor-zztpp","openshift-console/console-779db79f49-fqpnf","openshift-image-registry/node-ca-tfthv","openshift-multus/multus-tzlnj","openshift-multus/network-metrics-daemon-7bqg5","openshift-ovn-kubernetes/ovnkube-node-xgsjb","openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc","openshift-multus/multus-additional-cni-plugins-zkpxs","openshift-console/console-5fbc659c8d-gwbfw","openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m","openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv","openshift-dns/dns-default-9lr5s","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-n5gz6","openshift-monitoring/metrics-server-7dc8b9966c-xmpxj","openshift-network-operator/iptables-alerter-qtlzt","openshift-cluster-node-tuning-operator/tuned-k5ct6","openshift-monitoring/thanos-querier-7fd47999b4-r2bv9","openshift-monitoring/node-exporter-jdnjm","openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z","openshift-network-diagnostics/network-check-target-nnm9g","kube-system/konnectivity-agent-9fkqb","openshift-monitoring/alertmanager-main-0","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal","openshift-dns/node-resolver-rh2wd","openshift-ingress-canary/ingress-canary-4n77d"]
Oct 30 21:00:57.722677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.722615    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:00:57.722923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.722837    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.722955    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.722964    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-79n9v\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.722985    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gszwc\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723013    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723015    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jljmv\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723042    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.723054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723054    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723060    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723090    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723107    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723111    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723131    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723156    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4x2zw\""
Oct 30 21:00:57.723392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.723062    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:00:57.732628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.732611    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tzc2p\""
Oct 30 21:00:57.732737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.732722    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:00:57.733189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.733178    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:00:57.733411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.733398    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hvtl2\""
Oct 30 21:00:57.734711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.734696    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnqjk\""
Oct 30 21:00:57.734748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.734720    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7r6m\""
Oct 30 21:00:57.734788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.734773    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8v8x\""
Oct 30 21:00:57.735008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.734996    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fz2bf\""
Oct 30 21:00:57.735115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.735105    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-56hzc\""
Oct 30 21:00:57.735145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.735108    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:00:57.737511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.737483    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l57v4\""
Oct 30 21:00:57.737814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.737800    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8vzq\""
Oct 30 21:00:57.769160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.769138    7037 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:00:57.777075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.777049    7037 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.777188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.777138    7037 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.800270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.800251    7037 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:57.800352 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:57.800288    7037 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-72.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.800583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.800566    7037 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:57.800632 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:00:57.800606    7037 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-72.ec2.internal"
Oct 30 21:00:57.815288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815265    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.815373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815294    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.815373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815327    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.815373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815351    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.815516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815447    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.815516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815474    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkd4\" (UniqueName: \"kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.815606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815516    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.815606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815539    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.815606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.815606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815574    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.815606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815590    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm74\" (UniqueName: \"kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815622    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815697    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgh27\" (UniqueName: \"kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815718    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815732    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815747    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815778    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815803    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815849    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.815891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815873    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815897    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815933    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815962    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.815988    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816006    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816022    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816024    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovn-node-metrics-cert\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816064    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816122    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943e6093-44ff-4e3e-97d4-67a24ce560a0-tmp\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816169    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816198    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.816216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816219    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816239    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816266    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816309    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816336    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816360    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816445    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816465    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816481    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816522    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816583    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816607    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816614    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816636    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.816669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816669    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5v2k\" (UniqueName: \"kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816687    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7871fe29-fbbf-4733-878f-680fb56cee31-agent-certs\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816713    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxfr\" (UniqueName: \"kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816741    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816763    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816778    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816793    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816900    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816922    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816941    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816961    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.816998    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817070    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.817171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817139    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817177    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817208    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817228    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97x6l\" (UniqueName: \"kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817248    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817288    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817296    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817354    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817386    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2ea0220-eb35-4f3d-8147-060179390f3d-data-volume\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817436    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817468    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817533    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817538    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-tuned\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817594    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817613    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817651    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817672    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ff389-d912-4771-8a8f-43b6ab842829-metrics-certs\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:57.817707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817688    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817718    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bz8\" (UniqueName: \"kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817739    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxd4\" (UniqueName: \"kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817757    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817774    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zb2\" (UniqueName: \"kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817792    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817808    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817852    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817878    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817908    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817924    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817946    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817961    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.817979    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818002    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818001    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818097    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818119    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818136    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818158    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818185    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dp2\" (UniqueName: \"kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818211    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818222    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-tmp-dir\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818236    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818294    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.818321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818317    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-daemon-config\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818330    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818381    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818403    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bkz\" (UniqueName: \"kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818431    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818458    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ntk\" (UniqueName: \"kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818523    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818566    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818596    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bjx\" (UniqueName: \"kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818630    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818664    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2sb\" (UniqueName: \"kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818697    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818724    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818748    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818781    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818822    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818856    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-textfile\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818858    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818936    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818960    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm67s\" (UniqueName: \"kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818979    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.818997    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819037    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819131    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819153    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819172    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819189    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.819236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819222    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819253    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819279    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819306    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819334    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819353    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819369    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819400    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-tmp\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819421    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819442    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819458    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819481    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819534    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819589    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819609    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819627    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819646    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819663    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819682    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819697    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819735    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819779    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819812    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819835    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819858    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819877    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819896    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh7w\" (UniqueName: \"kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819957    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.819988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.819990    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820022    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm44\" (UniqueName: \"kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820048    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820076    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820106    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ce43bf79-f468-4826-ae78-e58f103f9fa7-audit-log\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820144    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820180    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820211    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820236    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820263    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820292    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820311    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820319    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggpw\" (UniqueName: \"kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820398    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c56952e8-b525-401d-8128-e4c8ac0d6469-serviceca\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820406    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820516    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820538    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820556    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820572    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820591    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820616    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820680    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820721    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820755    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820821    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820826    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820850    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820867    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820895    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820923    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.820976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820940    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.820993    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821018    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b36017a-ec01-4979-a358-7198261684d7-tmp-dir\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821027    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821057    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821083    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821112    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk8r\" (UniqueName: \"kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821158    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821197    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821236    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821247    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821270    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.821994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.821299    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.822835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.822815    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:00:57.822946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.822833    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:00:57.822946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.822846    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:00:57.823146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823129    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:00:57.823146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823150    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:00:57.823505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823467    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:00:57.823505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823478    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:00:57.823701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823680    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:00:57.823807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823789    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:00:57.823884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823795    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:00:57.823884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823795    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5jdx7\""
Oct 30 21:00:57.823884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823791    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:00:57.823884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823842    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:00:57.823884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823862    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823965    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-595gg\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.823971    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824036    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824037    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824063    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824067    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lm7tg\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824069    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vwnx\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824082    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824088    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824086    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5p0s1ifovnrri\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824117    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824127    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824132    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:00:57.824134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824069    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1c035iq9e8bv\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824305    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824339    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824348    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824352    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824366    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824585    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824603    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824654    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pdr56\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824663    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824678    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824688    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6gs8c\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824703    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824711    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824719    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824692    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824732    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824722    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824771    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jxvr7\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824785    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:00:57.824827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824788    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s9xpp\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824889    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-druc28d9l6vvr\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824895    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jqkvq\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824906    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824930    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.824984    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825033    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825036    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825081    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825125    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825135    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825295    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825179    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825177    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xstbj\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825196    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825197    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825226    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825311    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:00:57.825572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825207    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825907    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-metrics-tls\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.825939    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826007    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826055    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0bbf8b47-e869-43e4-8fdd-459639f068c3-original-pull-secret\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826081    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826120    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826246    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826246    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.826291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826269    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:00:57.826684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826389    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.826684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826249    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.826684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826430    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.826684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826524    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.826684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826658    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q65n4\""
Oct 30 21:00:57.826883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826679    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.826932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826876    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.826932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826913    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.827023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826933    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-client-certs\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.827023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.826947    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2nq7z\""
Oct 30 21:00:57.827161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827138    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.827212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827169    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943e6093-44ff-4e3e-97d4-67a24ce560a0-metrics-client-ca\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.827258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827235    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.827258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827241    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.827348 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827297    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.827395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827361    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.827524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827505    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.827577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827530    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.827577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827507    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-config-volume\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.827664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827586    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87e040ec-3520-40f4-b906-cc22de4d532a-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.827710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827660    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.827846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.827829    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-client-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.828055    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ce43bf79-f468-4826-ae78-e58f103f9fa7-secret-metrics-server-tls\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.828302    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/87e040ec-3520-40f4-b906-cc22de4d532a-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.828523    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.828699    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.828937    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.829162    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cni-binary-copy\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.829338    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.829538    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.829783    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.829989    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.830211    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.830434    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-config\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.830887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.830697    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831018    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831112    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831131    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831132    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-metrics-client-ca\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831270    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831059    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831347    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831356    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831443    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831452    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831407    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7871fe29-fbbf-4733-878f-680fb56cee31-konnectivity-ca\") pod \"konnectivity-agent-9fkqb\" (UID: \"7871fe29-fbbf-4733-878f-680fb56cee31\") " pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831463    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2ea0220-eb35-4f3d-8147-060179390f3d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831509    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.831530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831532    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831573    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831575    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831653    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831667    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-tls\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831731    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831735    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831786    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831766    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.831895    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-grpc-tls\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832020    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e300873-ed21-450b-9e6b-810ee569b783-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832058    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943e6093-44ff-4e3e-97d4-67a24ce560a0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.832151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832145    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:57.832650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832183    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.832650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832198    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.832650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.832308    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b088453c-eec1-465f-939e-e4c78cc2290b-cert\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:57.836195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836176    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836463    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836514    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836540    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836558    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836565    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836640    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836646    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:00:57.836697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836661    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.837029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836709    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.837029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836793    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.837029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836871    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.837029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.836917    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:00:57.837180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837152    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.837180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837165    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:00:57.837314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.837508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837474    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:00:57.837699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837683    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:00:57.837870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.837852    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-env-overrides\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.838157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.838139    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-accelerators-collector-config\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.838287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.838259    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.838968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.838944    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-metrics-server-audit-profiles\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.839097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839077    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:00:57.839827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839171    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkd4\" (UniqueName: \"kubernetes.io/projected/e76c52e5-2903-48df-92ea-bd07b934c773-kube-api-access-ckkd4\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.839827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839743    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgh27\" (UniqueName: \"kubernetes.io/projected/548cadb6-b5c0-4f7d-ad73-e5060de08fc9-kube-api-access-vgh27\") pod \"kube-state-metrics-5dc8d7d89b-r5n2m\" (UID: \"548cadb6-b5c0-4f7d-ad73-e5060de08fc9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5n2m"
Oct 30 21:00:57.839827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839760    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bz8\" (UniqueName: \"kubernetes.io/projected/1f1ff389-d912-4771-8a8f-43b6ab842829-kube-api-access-q5bz8\") pod \"network-metrics-daemon-7bqg5\" (UID: \"1f1ff389-d912-4771-8a8f-43b6ab842829\") " pod="openshift-multus/network-metrics-daemon-7bqg5"
Oct 30 21:00:57.839827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839788    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97x6l\" (UniqueName: \"kubernetes.io/projected/2b36017a-ec01-4979-a358-7198261684d7-kube-api-access-97x6l\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.840050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839884    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.840050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839896    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm74\" (UniqueName: \"kubernetes.io/projected/87e040ec-3520-40f4-b906-cc22de4d532a-kube-api-access-8nm74\") pod \"prometheus-operator-9cd89b74c-qmv5z\" (UID: \"87e040ec-3520-40f4-b906-cc22de4d532a\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qmv5z"
Oct 30 21:00:57.840050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.839934    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.840050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.840038    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5v2k\" (UniqueName: \"kubernetes.io/projected/e8062ab8-4d4e-41a1-9ed0-1328ffc23870-kube-api-access-g5v2k\") pod \"openshift-state-metrics-7bd7489957-p5f87\" (UID: \"e8062ab8-4d4e-41a1-9ed0-1328ffc23870\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p5f87"
Oct 30 21:00:57.840165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.840081    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxfr\" (UniqueName: \"kubernetes.io/projected/ce43bf79-f468-4826-ae78-e58f103f9fa7-kube-api-access-pgxfr\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.840165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.840110    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:00:57.840225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.840155    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-ovnkube-script-lib\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.840546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.840528    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:00:57.846663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.846639    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e76c52e5-2903-48df-92ea-bd07b934c773-iptables-alerter-script\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.846744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.846676    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce43bf79-f468-4826-ae78-e58f103f9fa7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dc8b9966c-xmpxj\" (UID: \"ce43bf79-f468-4826-ae78-e58f103f9fa7\") " pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:00:57.847000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.846983    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.849646    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.849715    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850132    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dp2\" (UniqueName: \"kubernetes.io/projected/c2ea0220-eb35-4f3d-8147-060179390f3d-kube-api-access-d2dp2\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850154    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850294    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh7w\" (UniqueName: \"kubernetes.io/projected/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-kube-api-access-2vh7w\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850305    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zb2\" (UniqueName: \"kubernetes.io/projected/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-kube-api-access-l2zb2\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850308    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850514    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzhk\" (UniqueName: \"kubernetes.io/projected/9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4-kube-api-access-qmzhk\") pod \"network-check-target-nnm9g\" (UID: \"9cabdea7-af0c-4c28-a3e4-d2ae7cce80d4\") " pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"console-779db79f49-fqpnf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") " pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850665    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxd4\" (UniqueName: \"kubernetes.io/projected/b088453c-eec1-465f-939e-e4c78cc2290b-kube-api-access-lhxd4\") pod \"ingress-canary-4n77d\" (UID: \"b088453c-eec1-465f-939e-e4c78cc2290b\") " pod="openshift-ingress-canary/ingress-canary-4n77d"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850889    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk8r\" (UniqueName: \"kubernetes.io/projected/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-kube-api-access-kqk8r\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850922    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bkz\" (UniqueName: \"kubernetes.io/projected/943e6093-44ff-4e3e-97d4-67a24ce560a0-kube-api-access-k2bkz\") pod \"thanos-querier-7fd47999b4-r2bv9\" (UID: \"943e6093-44ff-4e3e-97d4-67a24ce560a0\") " pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.850981    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm44\" (UniqueName: \"kubernetes.io/projected/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kube-api-access-thm44\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851056    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bjx\" (UniqueName: \"kubernetes.io/projected/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-kube-api-access-57bjx\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.851271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851193    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"console-5fbc659c8d-gwbfw\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") " pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:00:57.851940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851333    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm67s\" (UniqueName: \"kubernetes.io/projected/8211ee52-aa1e-4a0c-bfe8-3e1779da22a7-kube-api-access-wm67s\") pod \"monitoring-plugin-77f4f76594-cd2qc\" (UID: \"8211ee52-aa1e-4a0c-bfe8-3e1779da22a7\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:00:57.851940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851406    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggpw\" (UniqueName: \"kubernetes.io/projected/8e300873-ed21-450b-9e6b-810ee569b783-kube-api-access-6ggpw\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.851940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851622    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.851940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851636    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ntk\" (UniqueName: \"kubernetes.io/projected/1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c-kube-api-access-74ntk\") pod \"dns-default-9lr5s\" (UID: \"1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c\") " pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:00:57.851940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.851639    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2sb\" (UniqueName: \"kubernetes.io/projected/c56952e8-b525-401d-8128-e4c8ac0d6469-kube-api-access-jm2sb\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.854415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.854396    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.857591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.857572    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:00:57.921796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921739    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921771    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.921796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921789    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921811    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921834    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921845    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-node-log\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921867    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921881    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-multus\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921891    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921918    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-netns\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921926    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-conf-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921932    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921896    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-bin\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921946    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-conf\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921957    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921893    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-run\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921984    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.921999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.921992    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-etc-kubernetes\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922009    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922012    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922028    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-systemd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922028    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922044    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922060    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922076    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922092    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922106    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922120    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-system-cni-dir\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922119    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-socket-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922128    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922127    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922158    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-cni-bin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922162    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-kubernetes\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922166    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922159    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922206    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922236    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922255    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e300873-ed21-450b-9e6b-810ee569b783-cnibin\") pod \"multus-additional-cni-plugins-zkpxs\" (UID: \"8e300873-ed21-450b-9e6b-810ee569b783\") " pod="openshift-multus/multus-additional-cni-plugins-zkpxs"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922262    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922247    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-modprobe-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922293    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-hostroot\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922303    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922311    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2ea0220-eb35-4f3d-8147-060179390f3d-crio-socket\") pod \"insights-runtime-extractor-zztpp\" (UID: \"c2ea0220-eb35-4f3d-8147-060179390f3d\") " pod="openshift-insights/insights-runtime-extractor-zztpp"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922332    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-var-lib-kubelet\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922327    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922365    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922381    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922396    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922415    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-systemd-units\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922436    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922443    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-sys-fs\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922419    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922459    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-dbus\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922475    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-kubelet\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922484    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922530    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922546    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922562    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922587    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-system-cni-dir\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922591    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56952e8-b525-401d-8128-e4c8ac0d6469-host\") pod \"node-ca-tfthv\" (UID: \"c56952e8-b525-401d-8128-e4c8ac0d6469\") " pod="openshift-image-registry/node-ca-tfthv"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922600    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysconfig\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922606    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922627    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.922630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922636    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922663    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922682    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-log-socket\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922683    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-var-lib-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922700    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922725    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922755    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-run-ovn\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922766    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922777    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0bbf8b47-e869-43e4-8fdd-459639f068c3-kubelet-config\") pod \"global-pull-secret-syncer-n5gz6\" (UID: \"0bbf8b47-e869-43e4-8fdd-459639f068c3\") " pod="kube-system/global-pull-secret-syncer-n5gz6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922790    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922805    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922805    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-device-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922799    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-cnibin\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922842    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922844    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-sys\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922862    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-root\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922863    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922896    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-sysctl-d\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922896    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922941    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922966    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.922993    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923018    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923032    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-slash\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923043    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-sys\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923033    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-etc-openvswitch\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923042    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923082    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923031    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-host\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923109    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923128    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-netns\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923135    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923140    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593-registration-dir\") pod \"aws-ebs-csi-driver-node-k5xbv\" (UID: \"bbfbaf77-0475-4cc0-ad2c-3e1cfe49e593\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5xbv"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923143    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-var-lib-kubelet\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923166    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923187    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923203    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923218    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923228    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-lib-modules\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923232    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923253    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db-host-cni-netd\") pod \"ovnkube-node-xgsjb\" (UID: \"cebd2d0e-ac9b-4ffb-9da5-bb5cc6b391db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923265    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b36017a-ec01-4979-a358-7198261684d7-hosts-file\") pod \"node-resolver-rh2wd\" (UID: \"2b36017a-ec01-4979-a358-7198261684d7\") " pod="openshift-dns/node-resolver-rh2wd"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923270    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-multus-certs\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923278    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923289    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19-node-exporter-wtmp\") pod \"node-exporter-jdnjm\" (UID: \"6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19\") " pod="openshift-monitoring/node-exporter-jdnjm"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923296    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-multus-socket-dir-parent\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923301    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f-host-run-k8s-cni-cncf-io\") pod \"multus-tzlnj\" (UID: \"f60e19dc-42fc-4ad2-a73b-bb6f3bfc651f\") " pod="openshift-multus/multus-tzlnj"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923305    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923336    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923359    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5ac9b7b-1e1f-4966-8522-d92af9ee73b3-etc-systemd\") pod \"tuned-k5ct6\" (UID: \"b5ac9b7b-1e1f-4966-8522-d92af9ee73b3\") " pod="openshift-cluster-node-tuning-operator/tuned-k5ct6"
Oct 30 21:00:57.924168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:57.923446    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e76c52e5-2903-48df-92ea-bd07b934c773-host-slash\") pod \"iptables-alerter-qtlzt\" (UID: \"e76c52e5-2903-48df-92ea-bd07b934c773\") " pod="openshift-network-operator/iptables-alerter-qtlzt"
Oct 30 21:00:58.545697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:00:58.545663    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:01:00.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.277148    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:00.296603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.296576    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:00.326010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.325985    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9lr5s"
Oct 30 21:01:00.659825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.659798    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:01:00.660059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.660045    7037 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:00.829144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:00.829112    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9"
Oct 30 21:01:01.547737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:01.547700    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:01:02.514707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:02.514677    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:01:03.670718 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:03.670677    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:01:07.664233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:07.664149    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9fkqb"
Oct 30 21:01:07.974266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:07.974180    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:01:07.978742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:07.978717    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cd2qc"
Oct 30 21:01:11.431805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.431771    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:01:11.701920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.701827    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:11.702139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702113    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="alertmanager" containerID="cri-o://850fb538bf612204d79d236a210b091274501637dc3ddccc4417bafa620a6015" gracePeriod=120
Oct 30 21:01:11.702209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702180    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-web" containerID="cri-o://d8789f8b84d716526cc97428b87cbcf6be6067030687d76f4c49ae4023623c89" gracePeriod=120
Oct 30 21:01:11.702209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702208    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="config-reloader" containerID="cri-o://9b0e0a81703b0ccbdc596b2ff370db2913e1364f033d243f6059c890cbf5b61f" gracePeriod=120
Oct 30 21:01:11.702323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702223    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="prom-label-proxy" containerID="cri-o://6224f1d7cb08e8039afc519069589f90c1de09e4bd3e77d67fac145de4b34f06" gracePeriod=120
Oct 30 21:01:11.702323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702163    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-metric" containerID="cri-o://b10115285c74c45ca93512c5fd3b33191df94e749aa989979852a32d01b18984" gracePeriod=120
Oct 30 21:01:11.702323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.702213    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy" containerID="cri-o://7f9524e0428edc86db935ed9ec407b92bb603a968adfffbc86c5d49d90ffacdf" gracePeriod=120
Oct 30 21:01:11.805911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.805883    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgsjb"
Oct 30 21:01:11.835456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835424    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="6224f1d7cb08e8039afc519069589f90c1de09e4bd3e77d67fac145de4b34f06" exitCode=0
Oct 30 21:01:11.835456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835453    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="b10115285c74c45ca93512c5fd3b33191df94e749aa989979852a32d01b18984" exitCode=0
Oct 30 21:01:11.835625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835462    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="7f9524e0428edc86db935ed9ec407b92bb603a968adfffbc86c5d49d90ffacdf" exitCode=0
Oct 30 21:01:11.835625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835470    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="9b0e0a81703b0ccbdc596b2ff370db2913e1364f033d243f6059c890cbf5b61f" exitCode=0
Oct 30 21:01:11.835625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835477    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="850fb538bf612204d79d236a210b091274501637dc3ddccc4417bafa620a6015" exitCode=0
Oct 30 21:01:11.835625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835510    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"6224f1d7cb08e8039afc519069589f90c1de09e4bd3e77d67fac145de4b34f06"}
Oct 30 21:01:11.835625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835577    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"b10115285c74c45ca93512c5fd3b33191df94e749aa989979852a32d01b18984"}
Oct 30 21:01:11.835826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835643    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"7f9524e0428edc86db935ed9ec407b92bb603a968adfffbc86c5d49d90ffacdf"}
Oct 30 21:01:11.835826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835669    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"9b0e0a81703b0ccbdc596b2ff370db2913e1364f033d243f6059c890cbf5b61f"}
Oct 30 21:01:11.835826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:11.835690    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"850fb538bf612204d79d236a210b091274501637dc3ddccc4417bafa620a6015"}
Oct 30 21:01:12.842962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:12.842931    7037 generic.go:356] "Generic (PLEG): container finished" podID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerID="d8789f8b84d716526cc97428b87cbcf6be6067030687d76f4c49ae4023623c89" exitCode=0
Oct 30 21:01:12.843388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:12.843007    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"d8789f8b84d716526cc97428b87cbcf6be6067030687d76f4c49ae4023623c89"}
Oct 30 21:01:12.978392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:12.978365    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:13.072968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.072937    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.072989    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073057    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073088    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073132    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073165    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073214    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073263    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073292    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073325    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073372    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073407    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073441    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") pod \"2b802282-8c04-4522-8d63-1d3b66d897aa\" (UID: \"2b802282-8c04-4522-8d63-1d3b66d897aa\") "
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073453    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:13.073898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.073713    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-metrics-client-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.075280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.074859    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:13.077899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.076384    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:13.077899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.076977    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.078635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.078606    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.078999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.078973    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:13.079741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.079715    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:13.080061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.080032    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "kube-api-access-hlsx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:13.092042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.092009    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.092156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.092111    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.092229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.092184    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.092292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.092236    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.094306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.094278    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config" pod "2b802282-8c04-4522-8d63-1d3b66d897aa" (UID: "2b802282-8c04-4522-8d63-1d3b66d897aa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.175066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175026    7037 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175060    7037 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-web-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175070    7037 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-main-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175079    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsx4\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-kube-api-access-hlsx4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175089    7037 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175101    7037 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b802282-8c04-4522-8d63-1d3b66d897aa-tls-assets\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175111    7037 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175120    7037 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175128    7037 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-alertmanager-main-db\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175137    7037 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-cluster-tls-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175146    7037 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b802282-8c04-4522-8d63-1d3b66d897aa-config-out\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.175302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.175154    7037 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b802282-8c04-4522-8d63-1d3b66d897aa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:13.850460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.850421    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b802282-8c04-4522-8d63-1d3b66d897aa","Type":"ContainerDied","Data":"1db17f887334328e41be436515653e697ed1f71880a2e6b270fce3be556bf812"}
Oct 30 21:01:13.850460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.850464    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:13.850976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.850534    7037 scope.go:122] "RemoveContainer" containerID="6224f1d7cb08e8039afc519069589f90c1de09e4bd3e77d67fac145de4b34f06"
Oct 30 21:01:13.859507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.859463    7037 scope.go:122] "RemoveContainer" containerID="b10115285c74c45ca93512c5fd3b33191df94e749aa989979852a32d01b18984"
Oct 30 21:01:13.866773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.866756    7037 scope.go:122] "RemoveContainer" containerID="7f9524e0428edc86db935ed9ec407b92bb603a968adfffbc86c5d49d90ffacdf"
Oct 30 21:01:13.873644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.873606    7037 scope.go:122] "RemoveContainer" containerID="d8789f8b84d716526cc97428b87cbcf6be6067030687d76f4c49ae4023623c89"
Oct 30 21:01:13.884079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.884055    7037 scope.go:122] "RemoveContainer" containerID="9b0e0a81703b0ccbdc596b2ff370db2913e1364f033d243f6059c890cbf5b61f"
Oct 30 21:01:13.891647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.891629    7037 scope.go:122] "RemoveContainer" containerID="850fb538bf612204d79d236a210b091274501637dc3ddccc4417bafa620a6015"
Oct 30 21:01:13.899468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.899446    7037 scope.go:122] "RemoveContainer" containerID="d0159d09e37f3f934914c9cd69bce4ae05309ea35e90c490c3329541151d95f3"
Oct 30 21:01:13.921827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.921798    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:13.934510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.934474    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:13.956112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956083    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:13.956412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956400    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="config-reloader"
Oct 30 21:01:13.956455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956413    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="config-reloader"
Oct 30 21:01:13.956455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956429    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="alertmanager"
Oct 30 21:01:13.956455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956434    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="alertmanager"
Oct 30 21:01:13.956455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956441    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="init-config-reloader"
Oct 30 21:01:13.956455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956447    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="init-config-reloader"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956458    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956463    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956472    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="prom-label-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956477    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="prom-label-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956502    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-metric"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956508    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-metric"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956521    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-web"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956526    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-web"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956574    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="config-reloader"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956584    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="alertmanager"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956590    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="init-config-reloader"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956596    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956601    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="prom-label-proxy"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956608    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-metric"
Oct 30 21:01:13.956638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.956614    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" containerName="kube-rbac-proxy-web"
Oct 30 21:01:13.992182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.992153    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:13.992323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:13.992285    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.008857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.008824    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:01:14.011055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011030    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:01:14.011193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011086    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:01:14.011193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011109    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:01:14.011193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011187    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:01:14.011333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011228    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:01:14.011333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011262    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:01:14.011333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011306    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lm7tg\""
Oct 30 21:01:14.011333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.011330    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:01:14.024372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.024341    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:01:14.084566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084485    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084609    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084643    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084706    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-config-out\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084727    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-config-volume\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084787    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084809    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084825    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnk4w\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-kube-api-access-bnk4w\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084847    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-web-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084918    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084955    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.084999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084977    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.085089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.084995    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186269    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186306    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186330    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-config-out\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186349    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-config-volume\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186377    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186402    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186425    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnk4w\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-kube-api-access-bnk4w\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186455    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-web-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186482    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186529    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186559    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186590    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186643    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.186885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.186772    7037 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:01:14.187211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.187178    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.187465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.187435    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.187583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.187509    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db02ac6c-2f97-4944-b217-e48574269a40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.189450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.189421    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db02ac6c-2f97-4944-b217-e48574269a40-config-out\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.189719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.189691    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.189833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.189756    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.189833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.189787    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.189953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.189937    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-web-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.190040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.190017    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.190141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.190126    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.190252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.190233    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-config-volume\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.191238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.191220    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db02ac6c-2f97-4944-b217-e48574269a40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.206426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.206402    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnk4w\" (UniqueName: \"kubernetes.io/projected/db02ac6c-2f97-4944-b217-e48574269a40-kube-api-access-bnk4w\") pod \"alertmanager-main-0\" (UID: \"db02ac6c-2f97-4944-b217-e48574269a40\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.314253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.314219    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:01:14.629954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.629929    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:01:14.631207 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:01:14.631179    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb02ac6c_2f97_4944_b217_e48574269a40.slice/crio-7fc18f5fd279cd5c8e95b404bb31a5fb0d2ae6416a315d28daefda298a50a7bf WatchSource:0}: Error finding container 7fc18f5fd279cd5c8e95b404bb31a5fb0d2ae6416a315d28daefda298a50a7bf: Status 404 returned error can't find the container with id 7fc18f5fd279cd5c8e95b404bb31a5fb0d2ae6416a315d28daefda298a50a7bf
Oct 30 21:01:14.704906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.704878    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b802282-8c04-4522-8d63-1d3b66d897aa" path="/var/lib/kubelet/pods/2b802282-8c04-4522-8d63-1d3b66d897aa/volumes"
Oct 30 21:01:14.856131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.856101    7037 generic.go:356] "Generic (PLEG): container finished" podID="db02ac6c-2f97-4944-b217-e48574269a40" containerID="cd35a6a1e9d372f7173b3c94dc245bcba5e046f427e89f7e52a6fd875146d09d" exitCode=0
Oct 30 21:01:14.856541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.856137    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerDied","Data":"cd35a6a1e9d372f7173b3c94dc245bcba5e046f427e89f7e52a6fd875146d09d"}
Oct 30 21:01:14.856541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:14.856186    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"7fc18f5fd279cd5c8e95b404bb31a5fb0d2ae6416a315d28daefda298a50a7bf"}
Oct 30 21:01:15.864139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864102    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"9cb89c96773142d84d5c44373670580c1ce6a32505237fadd5552a5d2cea186c"}
Oct 30 21:01:15.864636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864189    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"ff2242770137eea438a159f186a4278a09afb004e7f96bf1dffd8d6c161c63a0"}
Oct 30 21:01:15.864636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864215    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"1d4f700627e6f894b2d76a7a1d22e1d6d2f8b5b552b1ff880f7935a1e81b7d68"}
Oct 30 21:01:15.864636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864240    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"403c61836708b9f8d5318e6a67385af717838f419f8d89c63666b82080c59a83"}
Oct 30 21:01:15.864636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864263    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"a4b8b9c2c936d90b71685daed58e0d7a5f0010db7f98d465a8a9de6d19688bbb"}
Oct 30 21:01:15.864636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.864287    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db02ac6c-2f97-4944-b217-e48574269a40","Type":"ContainerStarted","Data":"e140f74892d35460c138f31dc1ed06691a6673789a8175380bbf5850284547c5"}
Oct 30 21:01:15.893166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.893138    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:01:15.940224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:15.940159    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9281529280000003 podStartE2EDuration="2.940144373s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.991445ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:15.927079088 +0000 UTC m=+19.338618524" watchObservedRunningTime="2026-10-30 21:01:15.940144373 +0000 UTC m=+19.351683801"
Oct 30 21:01:16.289000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.288908    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289193    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy" containerID="cri-o://8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b" gracePeriod=600
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289193    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="prometheus" containerID="cri-o://db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767" gracePeriod=600
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289235    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="config-reloader" containerID="cri-o://18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4" gracePeriod=600
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289251    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="thanos-sidecar" containerID="cri-o://179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b" gracePeriod=600
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289242    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12" gracePeriod=600
Oct 30 21:01:16.289428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.289262    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-web" containerID="cri-o://5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b" gracePeriod=600
Oct 30 21:01:16.524881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.524856    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:16.609669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609629    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.609839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609682    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.609839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609709    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.609839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609746    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.609839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609796    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610195    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.610512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610450    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:16.610607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610472    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.610607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.609829    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610573    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610596    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610616    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610653    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610698    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610766    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.610814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610799    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610830    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610875    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610914    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.610956    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611001    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611034    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611067    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") pod \"453a3396-2bff-41c7-8164-25a8991c4e2d\" (UID: \"453a3396-2bff-41c7-8164-25a8991c4e2d\") "
Oct 30 21:01:16.611351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611278    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.611452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611391    7037 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.611452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611410    7037 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-metrics-client-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.611452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611426    7037 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-db\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.611452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.611441    7037 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.612556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.612050    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.612556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.612243    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.612556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.612445    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.612812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.612675    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.612874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.612844    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:16.613453 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.613418    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.613701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.613676    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.614088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.614064    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:16.615168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.615128    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.615396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.615369    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.615833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.615808    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.615914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.615868    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "kube-api-access-49dqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:16.616018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.615993    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.616320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.616292    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.616963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.616935    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:16.625032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.625009    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config" pod "453a3396-2bff-41c7-8164-25a8991c4e2d" (UID: "453a3396-2bff-41c7-8164-25a8991c4e2d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:16.712195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712157    7037 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712196    7037 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712213    7037 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712229    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dqm\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-kube-api-access-49dqm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712243    7037 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/453a3396-2bff-41c7-8164-25a8991c4e2d-config-out\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712257    7037 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-metrics-client-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712270    7037 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712283    7037 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-web-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712297    7037 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712306    7037 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/453a3396-2bff-41c7-8164-25a8991c4e2d-tls-assets\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712319    7037 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712332    7037 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712345    7037 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712354    7037 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-grpc-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712367    7037 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/453a3396-2bff-41c7-8164-25a8991c4e2d-secret-kube-rbac-proxy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.712440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.712381    7037 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/453a3396-2bff-41c7-8164-25a8991c4e2d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:16.870561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870465    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12" exitCode=0
Oct 30 21:01:16.870561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870511    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b" exitCode=0
Oct 30 21:01:16.870561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870524    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b" exitCode=0
Oct 30 21:01:16.870561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870532    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b" exitCode=0
Oct 30 21:01:16.870561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870531    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870585    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870599    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870540    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4" exitCode=0
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870627    7037 generic.go:356] "Generic (PLEG): container finished" podID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767" exitCode=0
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870640    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870677    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870705    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870721    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870751    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"}
Oct 30 21:01:16.871211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.870787    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"453a3396-2bff-41c7-8164-25a8991c4e2d","Type":"ContainerDied","Data":"2f6488c76e4332d07ed53e6e7bd6d39011fd2426d50f8c35dac6b287a96ab829"}
Oct 30 21:01:16.878167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.878140    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.885082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.885053    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.897631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.897611    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.906141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.906122    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.913499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.913460    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.921408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.921390    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.928012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.927986    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:16.929921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.929905    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.930218 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.930195    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.930273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930225    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.930273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930260    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.930548 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.930524    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.930596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930557    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.930596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930580    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.930823 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.930805    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.930858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930827    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.930858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.930841    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.931078 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.931063    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.931114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931081    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.931114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931094    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.931324 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.931309    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.931359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931331    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.931359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931346    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.931579 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.931559    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.931618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931587    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.931618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931606    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.931833 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:16.931816    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.931897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931834    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.931897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.931845    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.932081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932062    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.932125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932081    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.932307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932292    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.932348 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932308    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.932535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932519    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.932580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932534    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.932774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932756    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.932811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932774    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.933003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.932981    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.933003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933002    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.933202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933178    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.933245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933204    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.933408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933386    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.933449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933408    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.933629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933611    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.933688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933629    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.933888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933867    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.933958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.933889    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.934149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934130    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.934205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934149    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.934416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934381    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.934416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934416    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.934747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934722    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.934747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934746    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.934993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934968    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.934993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.934992    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.935244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935215    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.935448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935250    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.935530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935506    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.935593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935531    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.935841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935818    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.935841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.935839    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.936116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936098    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.936173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936117    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.936371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936351    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.936371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936370    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.936585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936569    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.936640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936585    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.936764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936749    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.936812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936765    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.936944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936928    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.936944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.936941    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.937106 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937092    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.937106 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937104    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.937289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937274    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.937334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937288    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.937472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937446    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.937540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937473    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.937684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937669    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.937726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937684    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.937880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937862    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.937916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.937881    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.938104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938087    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.938104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938103    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.938290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938275    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.938290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938289    7037 scope.go:122] "RemoveContainer" containerID="2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"
Oct 30 21:01:16.938528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938483    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12"} err="failed to get container status \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": rpc error: code = NotFound desc = could not find container \"2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12\": container with ID starting with 2b834911add666b03ec9220cdb2491041e73db7f54fefff39d8d567fda050d12 not found: ID does not exist"
Oct 30 21:01:16.938528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938519    7037 scope.go:122] "RemoveContainer" containerID="8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"
Oct 30 21:01:16.938696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938679    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b"} err="failed to get container status \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": rpc error: code = NotFound desc = could not find container \"8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b\": container with ID starting with 8d881ddeb55e3a7a34e17f769bbdc38d4dfe917ea3e393073aa5c382190b220b not found: ID does not exist"
Oct 30 21:01:16.938696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938695    7037 scope.go:122] "RemoveContainer" containerID="5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"
Oct 30 21:01:16.938880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938864    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b"} err="failed to get container status \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": rpc error: code = NotFound desc = could not find container \"5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b\": container with ID starting with 5b3ddc71b01c4fd0c899573deb752fee515b93021befc2eefbbb7167cde8f18b not found: ID does not exist"
Oct 30 21:01:16.938880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.938878    7037 scope.go:122] "RemoveContainer" containerID="179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"
Oct 30 21:01:16.939105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939083    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b"} err="failed to get container status \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": rpc error: code = NotFound desc = could not find container \"179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b\": container with ID starting with 179130c15552e9164c9393cde5aa24806c2c6b3f1ff0c49908d0c1dd7013776b not found: ID does not exist"
Oct 30 21:01:16.939105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939103    7037 scope.go:122] "RemoveContainer" containerID="18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"
Oct 30 21:01:16.939339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939314    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4"} err="failed to get container status \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": rpc error: code = NotFound desc = could not find container \"18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4\": container with ID starting with 18f1c7f6b7c5dc8f3fa0cd8c56f2461633f50f0d4dcbee40c4e47254c2346fd4 not found: ID does not exist"
Oct 30 21:01:16.939339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939332    7037 scope.go:122] "RemoveContainer" containerID="db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"
Oct 30 21:01:16.939556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939534    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767"} err="failed to get container status \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": rpc error: code = NotFound desc = could not find container \"db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767\": container with ID starting with db47d2072bf1b5d2a4e9a467ec529dd8ced732f3b92e6c398deb2105a7da9767 not found: ID does not exist"
Oct 30 21:01:16.939635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939557    7037 scope.go:122] "RemoveContainer" containerID="b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"
Oct 30 21:01:16.939797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.939775    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699"} err="failed to get container status \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": rpc error: code = NotFound desc = could not find container \"b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699\": container with ID starting with b0ff85e3612f9f860b76088a917d8f40f31f070f7bb4bf49ef4027a93dc79699 not found: ID does not exist"
Oct 30 21:01:16.940429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.940413    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:16.958853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.958827    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:16.959340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959319    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="thanos-sidecar"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959344    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="thanos-sidecar"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959369    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-web"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959375    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-web"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959383    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-thanos"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959389    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-thanos"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959397    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy"
Oct 30 21:01:16.959405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959403    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959415    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="prometheus"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959420    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="prometheus"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959429    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="init-config-reloader"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959434    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="init-config-reloader"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959443    7037 cpu_manager.go:414] "containerMap: removing container" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="config-reloader"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959447    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="config-reloader"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959511    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="thanos-sidecar"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959521    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-web"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959527    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy-thanos"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959532    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="kube-rbac-proxy"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959538    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="prometheus"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959543    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="init-config-reloader"
Oct 30 21:01:16.959649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.959548    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" containerName="config-reloader"
Oct 30 21:01:16.963110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.963095    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:16.980287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.980264    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:01:16.980422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.980286    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:01:16.980422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.980288    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:01:16.980618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.980592    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:01:16.980618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.980610    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:01:16.982765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.981029    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:01:16.982880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.982856    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:01:16.982983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.982899    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vwnx\""
Oct 30 21:01:16.982983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.982922    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:01:16.982983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.982934    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:01:16.983132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.982981    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:01:16.983132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.983021    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:01:16.983132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.983057    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:01:16.983251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.983219    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1c035iq9e8bv\""
Oct 30 21:01:16.996143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.996110    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:01:16.996279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.996191    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:01:16.996879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:16.996857    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:17.014859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.014831    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.014968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.014864    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.014968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.014886    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.014968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.014903    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.014987    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015069    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015106    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015126    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015141    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015183    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015237    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015261    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrf9\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-kube-api-access-5wrf9\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015283    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015303    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015351    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015400    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015428    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015474    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015530    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.015587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.015552    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116192    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116232    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116259    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116291    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116312    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116341    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116371    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.116476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.116399    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117009    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117075    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117114    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117110    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117147    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117200    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117228    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117252    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117296    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117344    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrf9\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-kube-api-access-5wrf9\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117405    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117439    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.117660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117471    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.118294    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.118441    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.118733    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.117203    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.119725    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.119950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.119926    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78e92c1a-1cea-405a-9691-6c490e6c8d32-config-out\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.120428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.120371    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.120817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.120753    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121023    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-web-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121280    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-config\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121338    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121649    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.121757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.121748    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.122525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.122483    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.122660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.122636    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/78e92c1a-1cea-405a-9691-6c490e6c8d32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.122920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.122901    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e92c1a-1cea-405a-9691-6c490e6c8d32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.137622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.137601    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrf9\" (UniqueName: \"kubernetes.io/projected/78e92c1a-1cea-405a-9691-6c490e6c8d32-kube-api-access-5wrf9\") pod \"prometheus-k8s-0\" (UID: \"78e92c1a-1cea-405a-9691-6c490e6c8d32\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.279952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.279912    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:17.598437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.598412    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:01:17.599600 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:01:17.599573    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e92c1a_1cea_405a_9691_6c490e6c8d32.slice/crio-3dfb7cadb8614f9086b1fc00c9255c51d44137b85cbe73b152f8d212dfc6ba19 WatchSource:0}: Error finding container 3dfb7cadb8614f9086b1fc00c9255c51d44137b85cbe73b152f8d212dfc6ba19: Status 404 returned error can't find the container with id 3dfb7cadb8614f9086b1fc00c9255c51d44137b85cbe73b152f8d212dfc6ba19
Oct 30 21:01:17.876421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.876326    7037 generic.go:356] "Generic (PLEG): container finished" podID="78e92c1a-1cea-405a-9691-6c490e6c8d32" containerID="2c18d3d1f7010b2ba55e8a35df8593d3a028e658249df9470f17572e78e89ec3" exitCode=0
Oct 30 21:01:17.876856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.876421    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerDied","Data":"2c18d3d1f7010b2ba55e8a35df8593d3a028e658249df9470f17572e78e89ec3"}
Oct 30 21:01:17.876856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:17.876570    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"3dfb7cadb8614f9086b1fc00c9255c51d44137b85cbe73b152f8d212dfc6ba19"}
Oct 30 21:01:18.704885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.704852    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453a3396-2bff-41c7-8164-25a8991c4e2d" path="/var/lib/kubelet/pods/453a3396-2bff-41c7-8164-25a8991c4e2d/volumes"
Oct 30 21:01:18.885351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885321    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"a54086b579fbd3e4ad1c2d41675bc98f381de7262b1c6b60be0cab42c08018af"}
Oct 30 21:01:18.885735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885384    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"8a217814e4baac012f53837e3514e8de884c23c865d88c436e5272e5f8a690b7"}
Oct 30 21:01:18.885735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885448    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"e12d95ee0f1c525b4348eed162fc32fdfc716a2d1baeb45081cff8934a926814"}
Oct 30 21:01:18.885735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885474    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"540037ca03e697d01927ae95cefcf80ef05409eea744a927809575c4c785a552"}
Oct 30 21:01:18.885735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885529    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"4e755f669db3e9da812b475e48600bc2b617d9048bcabe475d798f724476218d"}
Oct 30 21:01:18.885735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.885557    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"78e92c1a-1cea-405a-9691-6c490e6c8d32","Type":"ContainerStarted","Data":"b0a81eb09f386b4f243ad37aa9ab6b94d6f4509eb0b56a3d98ce09dc4d08f9b0"}
Oct 30 21:01:18.992101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:18.992004    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.976880114 podStartE2EDuration="2.991989618s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.109504ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:18.990793444 +0000 UTC m=+22.402332881" watchObservedRunningTime="2026-10-30 21:01:18.991989618 +0000 UTC m=+22.403529052"
Oct 30 21:01:22.280243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:22.280205    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:01:22.733629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:22.733561    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fbc659c8d-gwbfw" podUID="b23412f3-007f-4358-a2a1-09288ff60856" containerName="console" containerID="cri-o://ca4a09be5ce9c1a71ef99b2b19b8ada98743046220fc6115bb39c7c087f59a0b" gracePeriod=15
Oct 30 21:01:22.907667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:22.907638    7037 generic.go:356] "Generic (PLEG): container finished" podID="b23412f3-007f-4358-a2a1-09288ff60856" containerID="ca4a09be5ce9c1a71ef99b2b19b8ada98743046220fc6115bb39c7c087f59a0b" exitCode=0
Oct 30 21:01:22.907812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:22.907709    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbc659c8d-gwbfw" event={"ID":"b23412f3-007f-4358-a2a1-09288ff60856","Type":"ContainerDied","Data":"ca4a09be5ce9c1a71ef99b2b19b8ada98743046220fc6115bb39c7c087f59a0b"}
Oct 30 21:01:23.001377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.001318    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:01:23.076783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.076756    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.076932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.076792    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.076932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.076831    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077042    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:23.077118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077091    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077139    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077180    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077233    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077273    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") pod \"b23412f3-007f-4358-a2a1-09288ff60856\" (UID: \"b23412f3-007f-4358-a2a1-09288ff60856\") "
Oct 30 21:01:23.077696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077543    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:23.077696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077571    7037 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23412f3-007f-4358-a2a1-09288ff60856-tmp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.077910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077750    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:23.077910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077803    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:23.077910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.077845    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:23.079305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.079279    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:23.079402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.079350    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "kube-api-access-2pnzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:23.079402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.079358    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert" pod "b23412f3-007f-4358-a2a1-09288ff60856" (UID: "b23412f3-007f-4358-a2a1-09288ff60856"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:23.178812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178782    7037 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-oauth-serving-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.178812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178806    7037 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.178812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178816    7037 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-serving-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.179018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178826    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnzh\" (UniqueName: \"kubernetes.io/projected/b23412f3-007f-4358-a2a1-09288ff60856-kube-api-access-2pnzh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.179018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178836    7037 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23412f3-007f-4358-a2a1-09288ff60856-console-oauth-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.179018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178845    7037 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-console-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.179018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.178853    7037 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23412f3-007f-4358-a2a1-09288ff60856-trusted-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:23.913219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.913188    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbc659c8d-gwbfw"
Oct 30 21:01:23.913626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.913215    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbc659c8d-gwbfw" event={"ID":"b23412f3-007f-4358-a2a1-09288ff60856","Type":"ContainerDied","Data":"38c26875765559af99407f7566cd262d7712f9e0162da483a1ad2ad1f612b300"}
Oct 30 21:01:23.913626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.913276    7037 scope.go:122] "RemoveContainer" containerID="ca4a09be5ce9c1a71ef99b2b19b8ada98743046220fc6115bb39c7c087f59a0b"
Oct 30 21:01:23.968584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.968546    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:01:23.980784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:23.980751    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fbc659c8d-gwbfw"]
Oct 30 21:01:24.709171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:24.705785    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23412f3-007f-4358-a2a1-09288ff60856" path="/var/lib/kubelet/pods/b23412f3-007f-4358-a2a1-09288ff60856/volumes"
Oct 30 21:01:31.436454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:31.436423    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:01:31.440040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:31.440017    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7dc8b9966c-xmpxj"
Oct 30 21:01:31.550206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:31.550179    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nnm9g"
Oct 30 21:01:31.787841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:31.787767    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:01:33.671059 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:01:33.671021    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:01:56.807345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:56.807305    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-779db79f49-fqpnf" podUID="826b8c4d-26f6-47f8-a885-85c93b1996cf" containerName="console" containerID="cri-o://d41f6145e7fa093d953c3c92d6af819e0d462f2067d2e4d5ff35972bcfbc2caf" gracePeriod=15
Oct 30 21:01:57.048097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.048065    7037 generic.go:356] "Generic (PLEG): container finished" podID="826b8c4d-26f6-47f8-a885-85c93b1996cf" containerID="d41f6145e7fa093d953c3c92d6af819e0d462f2067d2e4d5ff35972bcfbc2caf" exitCode=0
Oct 30 21:01:57.048220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.048137    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779db79f49-fqpnf" event={"ID":"826b8c4d-26f6-47f8-a885-85c93b1996cf","Type":"ContainerDied","Data":"d41f6145e7fa093d953c3c92d6af819e0d462f2067d2e4d5ff35972bcfbc2caf"}
Oct 30 21:01:57.048220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.048188    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779db79f49-fqpnf" event={"ID":"826b8c4d-26f6-47f8-a885-85c93b1996cf","Type":"ContainerDied","Data":"8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68"}
Oct 30 21:01:57.048220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.048202    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8127bfb6ceb6ad11c4326c84600b7a123b03ed3f2eedf81c5260e8691b839c68"
Oct 30 21:01:57.053483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.053465    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:01:57.172796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.172741    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.172796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.172814    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.172847    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.172940    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173016    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173059    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173089    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173115    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") pod \"826b8c4d-26f6-47f8-a885-85c93b1996cf\" (UID: \"826b8c4d-26f6-47f8-a885-85c93b1996cf\") "
Oct 30 21:01:57.173364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173343    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:57.173415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173283    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:57.173474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173452    7037 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-oauth-serving-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.173474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173454    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:57.173593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173470    7037 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826b8c4d-26f6-47f8-a885-85c93b1996cf-tmp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.173593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173469    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:57.173716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.173684    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:57.175133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.175109    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:57.175239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.175150    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:57.175298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.175248    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d" pod "826b8c4d-26f6-47f8-a885-85c93b1996cf" (UID: "826b8c4d-26f6-47f8-a885-85c93b1996cf"). InnerVolumeSpecName "kube-api-access-s5f8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274765    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5f8d\" (UniqueName: \"kubernetes.io/projected/826b8c4d-26f6-47f8-a885-85c93b1996cf-kube-api-access-s5f8d\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274810    7037 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274822    7037 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-trusted-ca-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274830    7037 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-serving-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274839    7037 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:57.274865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:57.274847    7037 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/826b8c4d-26f6-47f8-a885-85c93b1996cf-console-oauth-config\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:01:58.052267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:58.052234    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779db79f49-fqpnf"
Oct 30 21:01:58.104874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:58.104844    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:01:58.115107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:58.115071    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-779db79f49-fqpnf"]
Oct 30 21:01:58.704599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:01:58.704567    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826b8c4d-26f6-47f8-a885-85c93b1996cf" path="/var/lib/kubelet/pods/826b8c4d-26f6-47f8-a885-85c93b1996cf/volumes"
Oct 30 21:02:03.669981 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:02:03.669946    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:02:12.237810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:02:12.237772    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:02:12.253310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:02:12.253285    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:02:12.296254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:02:12.296231    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:02:33.670451 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:02:33.670410    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:03:03.670196 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:03:03.670110    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:03:33.670077 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:03:33.670034    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:04:02.693182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693144    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"]
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693533    7037 cpu_manager.go:414] "containerMap: removing container" podUID="826b8c4d-26f6-47f8-a885-85c93b1996cf" containerName="console"
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693545    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826b8c4d-26f6-47f8-a885-85c93b1996cf" containerName="console"
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693580    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b23412f3-007f-4358-a2a1-09288ff60856" containerName="console"
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693586    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23412f3-007f-4358-a2a1-09288ff60856" containerName="console"
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693637    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="826b8c4d-26f6-47f8-a885-85c93b1996cf" containerName="console"
Oct 30 21:04:02.693761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.693651    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23412f3-007f-4358-a2a1-09288ff60856" containerName="console"
Oct 30 21:04:02.696819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.696803    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.715294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.715264    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9k4\" (UniqueName: \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.715481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.715330    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.715481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.715388    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.724119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.724091    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:04:02.724268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.724169    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:04:02.724337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.724316    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vplgz\""
Oct 30 21:04:02.725019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.724995    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"]
Oct 30 21:04:02.816151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.816099    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.816334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.816184    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9k4\" (UniqueName: \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.816334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.816222    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.816623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.816607    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.816666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.816604    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:02.835951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:02.835927    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9k4\" (UniqueName: \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:03.012656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:03.012573    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:03.306157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:03.306130    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"]
Oct 30 21:04:03.307430 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:04:03.307405    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6040637b_85e3_4cee_9efb_ddc03da450df.slice/crio-6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895 WatchSource:0}: Error finding container 6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895: Status 404 returned error can't find the container with id 6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895
Oct 30 21:04:03.309479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:03.309456    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:04:03.521618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:03.521579    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd" event={"ID":"6040637b-85e3-4cee-9efb-ddc03da450df","Type":"ContainerStarted","Data":"6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895"}
Oct 30 21:04:03.674925 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:04:03.674887    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:04:09.552933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:09.552893    7037 generic.go:356] "Generic (PLEG): container finished" podID="6040637b-85e3-4cee-9efb-ddc03da450df" containerID="afd244aa5272f5069259d39b34dff146161688cc68d026a175f28f339b9d551c" exitCode=0
Oct 30 21:04:09.553464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:09.552962    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd" event={"ID":"6040637b-85e3-4cee-9efb-ddc03da450df","Type":"ContainerDied","Data":"afd244aa5272f5069259d39b34dff146161688cc68d026a175f28f339b9d551c"}
Oct 30 21:04:11.565135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:11.565095    7037 generic.go:356] "Generic (PLEG): container finished" podID="6040637b-85e3-4cee-9efb-ddc03da450df" containerID="12072040d31b19e01478398d03893b5456a22ac4505f1222f36da7e4ddecc505" exitCode=0
Oct 30 21:04:11.565513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:11.565160    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd" event={"ID":"6040637b-85e3-4cee-9efb-ddc03da450df","Type":"ContainerDied","Data":"12072040d31b19e01478398d03893b5456a22ac4505f1222f36da7e4ddecc505"}
Oct 30 21:04:17.591394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:17.591359    7037 generic.go:356] "Generic (PLEG): container finished" podID="6040637b-85e3-4cee-9efb-ddc03da450df" containerID="b3561f7ae2fd85fd2e9094d445877cf23b1e4c1058deae00dadeec51979772c6" exitCode=0
Oct 30 21:04:17.591803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:17.591436    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd" event={"ID":"6040637b-85e3-4cee-9efb-ddc03da450df","Type":"ContainerDied","Data":"b3561f7ae2fd85fd2e9094d445877cf23b1e4c1058deae00dadeec51979772c6"}
Oct 30 21:04:18.717663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.717627    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:18.763527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.763476    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\" (UniqueName: \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\") pod \"6040637b-85e3-4cee-9efb-ddc03da450df\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") "
Oct 30 21:04:18.763670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.763544    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\") pod \"6040637b-85e3-4cee-9efb-ddc03da450df\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") "
Oct 30 21:04:18.763670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.763646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\") pod \"6040637b-85e3-4cee-9efb-ddc03da450df\" (UID: \"6040637b-85e3-4cee-9efb-ddc03da450df\") "
Oct 30 21:04:18.764033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.763999    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle" pod "6040637b-85e3-4cee-9efb-ddc03da450df" (UID: "6040637b-85e3-4cee-9efb-ddc03da450df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:18.768629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.768594    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4" pod "6040637b-85e3-4cee-9efb-ddc03da450df" (UID: "6040637b-85e3-4cee-9efb-ddc03da450df"). InnerVolumeSpecName "kube-api-access-sx9k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:18.769686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.769655    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util" pod "6040637b-85e3-4cee-9efb-ddc03da450df" (UID: "6040637b-85e3-4cee-9efb-ddc03da450df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:18.865064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.864994    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9k4\" (UniqueName: \"kubernetes.io/projected/6040637b-85e3-4cee-9efb-ddc03da450df-kube-api-access-sx9k4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:04:18.865064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.865019    7037 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-util\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:04:18.865064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:18.865030    7037 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6040637b-85e3-4cee-9efb-ddc03da450df-bundle\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:04:19.601024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:19.600997    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd"
Oct 30 21:04:19.601185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:19.600984    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wwxbd" event={"ID":"6040637b-85e3-4cee-9efb-ddc03da450df","Type":"ContainerDied","Data":"6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895"}
Oct 30 21:04:19.601185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:19.601128    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6461e8ccd9b3ddc70274c6c12d6d84ff7feaeb1fb69929e7f6f9871db33fc895"
Oct 30 21:04:24.658137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658094    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"]
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658566    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="util"
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658585    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="util"
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658617    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="extract"
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658624    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="extract"
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658653    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="pull"
Oct 30 21:04:24.658672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658660    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="pull"
Oct 30 21:04:24.658961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.658742    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6040637b-85e3-4cee-9efb-ddc03da450df" containerName="extract"
Oct 30 21:04:24.660820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.660799    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.675747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.675724    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:04:24.675747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.675734    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:04:24.688709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.688686    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-nn974\""
Oct 30 21:04:24.689616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.689593    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"]
Oct 30 21:04:24.715764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.715739    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fpr\" (UniqueName: \"kubernetes.io/projected/d0d505fa-b7f7-4684-9631-6f02b061c2dc-kube-api-access-g2fpr\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.715917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.715778    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d505fa-b7f7-4684-9631-6f02b061c2dc-tmp\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.816802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.816770    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fpr\" (UniqueName: \"kubernetes.io/projected/d0d505fa-b7f7-4684-9631-6f02b061c2dc-kube-api-access-g2fpr\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.816972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.816810    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d505fa-b7f7-4684-9631-6f02b061c2dc-tmp\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.817181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.817164    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0d505fa-b7f7-4684-9631-6f02b061c2dc-tmp\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.835691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.835660    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fpr\" (UniqueName: \"kubernetes.io/projected/d0d505fa-b7f7-4684-9631-6f02b061c2dc-kube-api-access-g2fpr\") pod \"jobset-operator-674cd59dc5-xttrb\" (UID: \"d0d505fa-b7f7-4684-9631-6f02b061c2dc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:24.990534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:24.990441    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"
Oct 30 21:04:25.277256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:25.277225    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb"]
Oct 30 21:04:25.278578 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:04:25.278547    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d505fa_b7f7_4684_9631_6f02b061c2dc.slice/crio-7e650772314bde85f625cc83d26a4a3aec603cf575f6b325d98f668478b00a0b WatchSource:0}: Error finding container 7e650772314bde85f625cc83d26a4a3aec603cf575f6b325d98f668478b00a0b: Status 404 returned error can't find the container with id 7e650772314bde85f625cc83d26a4a3aec603cf575f6b325d98f668478b00a0b
Oct 30 21:04:25.624905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:25.624869    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb" event={"ID":"d0d505fa-b7f7-4684-9631-6f02b061c2dc","Type":"ContainerStarted","Data":"7e650772314bde85f625cc83d26a4a3aec603cf575f6b325d98f668478b00a0b"}
Oct 30 21:04:27.635256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:27.635215    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb" event={"ID":"d0d505fa-b7f7-4684-9631-6f02b061c2dc","Type":"ContainerStarted","Data":"45f808bf7a8eb423c31265dcc789e8579602d0d046b974440606c943e43ad016"}
Oct 30 21:04:27.681433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:27.681374    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xttrb" podStartSLOduration=1.723841974 podStartE2EDuration="3.681360991s" totalImagesPullingTime="1.957519017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:27.680928149 +0000 UTC m=+211.092467586" watchObservedRunningTime="2026-10-30 21:04:27.681360991 +0000 UTC m=+211.092900420"
Oct 30 21:04:33.670248 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:04:33.670211    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:04:35.156957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.156923    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"]
Oct 30 21:04:35.159315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.159299    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.190898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.190859    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"]
Oct 30 21:04:35.269009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.268972    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:04:35.269205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.269139    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:04:35.269302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.269281    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t56h2\""
Oct 30 21:04:35.269353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.269337    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:04:35.272888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.272873    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:04:35.272975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.272964    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:04:35.306336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306310    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.306336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306338    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.306523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306358    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.306571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306555    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.306606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306590    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9h8r\" (UniqueName: \"kubernetes.io/projected/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-kube-api-access-p9h8r\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.306642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.306609    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.407889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407794    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.407889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407837    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9h8r\" (UniqueName: \"kubernetes.io/projected/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-kube-api-access-p9h8r\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.407889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407856    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.407889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407883    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.408211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407904    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.408211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.407929    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.408324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.408299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.410280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.410252    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.410692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.410667    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.410783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.410691    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.410783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.410667    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.433041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.433011    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9h8r\" (UniqueName: \"kubernetes.io/projected/5432da5c-311f-4f6a-9bcc-5e15db94b4a1-kube-api-access-p9h8r\") pod \"opendatahub-operator-controller-manager-98586cfbd-kw7lw\" (UID: \"5432da5c-311f-4f6a-9bcc-5e15db94b4a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.472666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.472631    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:35.804372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:35.804343    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"]
Oct 30 21:04:35.804961 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:04:35.804936    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5432da5c_311f_4f6a_9bcc_5e15db94b4a1.slice/crio-ba095e3fd503dcfe0769bca680e529f235fe0e005b12d59afb070c7b21e8c087 WatchSource:0}: Error finding container ba095e3fd503dcfe0769bca680e529f235fe0e005b12d59afb070c7b21e8c087: Status 404 returned error can't find the container with id ba095e3fd503dcfe0769bca680e529f235fe0e005b12d59afb070c7b21e8c087
Oct 30 21:04:36.671247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:36.671205    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw" event={"ID":"5432da5c-311f-4f6a-9bcc-5e15db94b4a1","Type":"ContainerStarted","Data":"ba095e3fd503dcfe0769bca680e529f235fe0e005b12d59afb070c7b21e8c087"}
Oct 30 21:04:39.690972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:39.690877    7037 generic.go:356] "Generic (PLEG): container finished" podID="5432da5c-311f-4f6a-9bcc-5e15db94b4a1" containerID="d8bb0bfde359260f3e50bba6883e08462c45f33dfd9b4fd71058d28df6545f5c" exitCode=0
Oct 30 21:04:39.690972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:39.690956    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw" event={"ID":"5432da5c-311f-4f6a-9bcc-5e15db94b4a1","Type":"ContainerDied","Data":"d8bb0bfde359260f3e50bba6883e08462c45f33dfd9b4fd71058d28df6545f5c"}
Oct 30 21:04:40.697056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:40.697019    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw" event={"ID":"5432da5c-311f-4f6a-9bcc-5e15db94b4a1","Type":"ContainerStarted","Data":"444e2592205b550457ff78fd2c2251f0d4c27dd8aa5afff68e785790b26453d8"}
Oct 30 21:04:40.697472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:40.697180    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:40.850646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:40.850591    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw" podStartSLOduration=2.008461621 podStartE2EDuration="5.850576076s" totalImagesPullingTime="3.268425969s" totalInitContainerRuntime="573.688486ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:40.838592349 +0000 UTC m=+224.250131788" watchObservedRunningTime="2026-10-30 21:04:40.850576076 +0000 UTC m=+224.262115507"
Oct 30 21:04:48.442542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.442502    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"]
Oct 30 21:04:48.493214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.493167    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"]
Oct 30 21:04:48.493295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.493270    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.508389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.508368    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:04:48.508518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.508448    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:04:48.556538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.556512    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fzmh2\""
Oct 30 21:04:48.623621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.623586    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.623780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.623639    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6qv\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-kube-api-access-2f6qv\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.724456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.724380    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.724456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.724428    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6qv\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-kube-api-access-2f6qv\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.748505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.748471    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.748693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.748672    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6qv\" (UniqueName: \"kubernetes.io/projected/8a64e2b4-d8fd-487c-8220-cb56ad0e0df3-kube-api-access-2f6qv\") pod \"cert-manager-cainjector-5c57bbd4fc-9r4rm\" (UID: \"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:48.825307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:48.825269    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"
Oct 30 21:04:49.139029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:49.139003    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm"]
Oct 30 21:04:49.140290 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:04:49.140255    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a64e2b4_d8fd_487c_8220_cb56ad0e0df3.slice/crio-b9b4a42ab4f6a6d6f0d5e62107104320f6889d983327ac86377789a27c294c8c WatchSource:0}: Error finding container b9b4a42ab4f6a6d6f0d5e62107104320f6889d983327ac86377789a27c294c8c: Status 404 returned error can't find the container with id b9b4a42ab4f6a6d6f0d5e62107104320f6889d983327ac86377789a27c294c8c
Oct 30 21:04:49.737948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:49.737908    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm" event={"ID":"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3","Type":"ContainerStarted","Data":"b9b4a42ab4f6a6d6f0d5e62107104320f6889d983327ac86377789a27c294c8c"}
Oct 30 21:04:51.703037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:51.703000    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-kw7lw"
Oct 30 21:04:52.753742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:52.753705    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm" event={"ID":"8a64e2b4-d8fd-487c-8220-cb56ad0e0df3","Type":"ContainerStarted","Data":"daf71837dde6aee6d8f1f5d7e64b0b9c67e98da5ed56eafebdeab671c9a53aa5"}
Oct 30 21:04:52.798034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:52.797985    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9r4rm" podStartSLOduration=1.788717422 podStartE2EDuration="4.797971155s" totalImagesPullingTime="3.009253733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:52.797265074 +0000 UTC m=+236.208804511" watchObservedRunningTime="2026-10-30 21:04:52.797971155 +0000 UTC m=+236.209510570"
Oct 30 21:04:57.383578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.383539    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b45l6"]
Oct 30 21:04:57.386466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.386448    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.395993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.395971    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwl9\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-kube-api-access-8hwl9\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.396109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.396029    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-bound-sa-token\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.417469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.417442    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dlz9x\""
Oct 30 21:04:57.418606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.418585    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b45l6"]
Oct 30 21:04:57.496782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.496752    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-bound-sa-token\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.496951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.496811    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwl9\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-kube-api-access-8hwl9\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.516743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.516711    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwl9\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-kube-api-access-8hwl9\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.516861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.516842    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d4e4696-5577-45f2-ad90-0fd53c2548a0-bound-sa-token\") pod \"cert-manager-597b95cb84-b45l6\" (UID: \"0d4e4696-5577-45f2-ad90-0fd53c2548a0\") " pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.696648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.696570    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b45l6"
Oct 30 21:04:57.987595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:57.987570    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b45l6"]
Oct 30 21:04:57.989128 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:04:57.989100    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4e4696_5577_45f2_ad90_0fd53c2548a0.slice/crio-c0895d6e33ceca7b82c5f1d9eae8088d90c9d27562e0083cfdc1045b32768c17 WatchSource:0}: Error finding container c0895d6e33ceca7b82c5f1d9eae8088d90c9d27562e0083cfdc1045b32768c17: Status 404 returned error can't find the container with id c0895d6e33ceca7b82c5f1d9eae8088d90c9d27562e0083cfdc1045b32768c17
Oct 30 21:04:58.779358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:58.779316    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b45l6" event={"ID":"0d4e4696-5577-45f2-ad90-0fd53c2548a0","Type":"ContainerStarted","Data":"aa27dcc863df4ec8613c31214f7251d9ba69ac96115c778152c06eaacfe62006"}
Oct 30 21:04:58.779774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:58.779375    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b45l6" event={"ID":"0d4e4696-5577-45f2-ad90-0fd53c2548a0","Type":"ContainerStarted","Data":"c0895d6e33ceca7b82c5f1d9eae8088d90c9d27562e0083cfdc1045b32768c17"}
Oct 30 21:04:58.823099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:04:58.823048    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b45l6" podStartSLOduration=1.823034536 podStartE2EDuration="1.823034536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:58.822454384 +0000 UTC m=+242.233993821" watchObservedRunningTime="2026-10-30 21:04:58.823034536 +0000 UTC m=+242.234573973"
Oct 30 21:05:03.672130 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:05:03.672091    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:05:08.623223 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.623182    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"]
Oct 30 21:05:08.632774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.632745    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.644555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.644525    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"]
Oct 30 21:05:08.655862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.655832    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6kfs6\""
Oct 30 21:05:08.656181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.656162    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:05:08.666673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.666650    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:05:08.666673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.666653    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:05:08.693310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.693288    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:05:08.699699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.699675    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79nr\" (UniqueName: \"kubernetes.io/projected/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-kube-api-access-z79nr\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.699806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.699709    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tmp\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.699881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.699830    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tls-certs\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.800622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.800585    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tls-certs\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.800775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.800639    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79nr\" (UniqueName: \"kubernetes.io/projected/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-kube-api-access-z79nr\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.800775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.800669    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tmp\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.802936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.802899    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tmp\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.803119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.803096    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-tls-certs\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.828293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.828261    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79nr\" (UniqueName: \"kubernetes.io/projected/aab318c8-8afe-4e74-b82e-a85e5fd1e59a-kube-api-access-z79nr\") pod \"kube-auth-proxy-5649f7bc66-trj7m\" (UID: \"aab318c8-8afe-4e74-b82e-a85e5fd1e59a\") " pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:08.946328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:08.946241    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"
Oct 30 21:05:09.238162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.238116    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m"]
Oct 30 21:05:09.238736 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:05:09.238696    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab318c8_8afe_4e74_b82e_a85e5fd1e59a.slice/crio-9ddc05b1ea0c785222a1bac21053a2abf403746cc42cfb4e5276ce128e7f68ae WatchSource:0}: Error finding container 9ddc05b1ea0c785222a1bac21053a2abf403746cc42cfb4e5276ce128e7f68ae: Status 404 returned error can't find the container with id 9ddc05b1ea0c785222a1bac21053a2abf403746cc42cfb4e5276ce128e7f68ae
Oct 30 21:05:09.761059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.761028    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"]
Oct 30 21:05:09.766006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.765982    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.783648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.783620    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:05:09.795688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.795660    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:05:09.795688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.795660    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:05:09.795688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.795680    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:05:09.796038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.795882    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lmssf\""
Oct 30 21:05:09.797458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.797436    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"]
Oct 30 21:05:09.808856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.808804    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vd2c\" (UniqueName: \"kubernetes.io/projected/249f3602-e4df-406a-ad76-f1e605158d4d-kube-api-access-2vd2c\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.808985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.808879    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/249f3602-e4df-406a-ad76-f1e605158d4d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.808985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.808925    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/249f3602-e4df-406a-ad76-f1e605158d4d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.824661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.824629    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m" event={"ID":"aab318c8-8afe-4e74-b82e-a85e5fd1e59a","Type":"ContainerStarted","Data":"9ddc05b1ea0c785222a1bac21053a2abf403746cc42cfb4e5276ce128e7f68ae"}
Oct 30 21:05:09.909843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.909797    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vd2c\" (UniqueName: \"kubernetes.io/projected/249f3602-e4df-406a-ad76-f1e605158d4d-kube-api-access-2vd2c\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.910014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.909866    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/249f3602-e4df-406a-ad76-f1e605158d4d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.910014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.909902    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/249f3602-e4df-406a-ad76-f1e605158d4d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.910940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.910907    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/249f3602-e4df-406a-ad76-f1e605158d4d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.913000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.912967    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/249f3602-e4df-406a-ad76-f1e605158d4d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:09.948440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:09.948404    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vd2c\" (UniqueName: \"kubernetes.io/projected/249f3602-e4df-406a-ad76-f1e605158d4d-kube-api-access-2vd2c\") pod \"kuadrant-console-plugin-66c4fb575d-w87qg\" (UID: \"249f3602-e4df-406a-ad76-f1e605158d4d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:10.079571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.079538    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"
Oct 30 21:05:10.383369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.383339    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg"]
Oct 30 21:05:10.384511 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:05:10.384474    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod249f3602_e4df_406a_ad76_f1e605158d4d.slice/crio-df60c8c7fceaf83db81309799f4a75578812a9d53cd254675a5da487decb2fb2 WatchSource:0}: Error finding container df60c8c7fceaf83db81309799f4a75578812a9d53cd254675a5da487decb2fb2: Status 404 returned error can't find the container with id df60c8c7fceaf83db81309799f4a75578812a9d53cd254675a5da487decb2fb2
Oct 30 21:05:10.715609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.715528    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:10.722470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.722438    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:10.742584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.742553    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:10.747878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.747847    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5wzgz\""
Oct 30 21:05:10.819190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.819157    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\") pod \"authorino-57f7f8659f-l7bhb\" (UID: \"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659\") " pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:10.831362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.831327    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg" event={"ID":"249f3602-e4df-406a-ad76-f1e605158d4d","Type":"ContainerStarted","Data":"df60c8c7fceaf83db81309799f4a75578812a9d53cd254675a5da487decb2fb2"}
Oct 30 21:05:10.920586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.920548    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\") pod \"authorino-57f7f8659f-l7bhb\" (UID: \"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659\") " pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:10.957081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:10.950964    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\") pod \"authorino-57f7f8659f-l7bhb\" (UID: \"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659\") " pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:11.038001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.037918    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:11.362561 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:05:11.362515    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ab3a2b5_2db2_43d9_bdf8_59826bcd0659.slice/crio-2337b185867e772eb4066472546e8abc5983fe1e26a38c3ed4b5ecc73cb47691 WatchSource:0}: Error finding container 2337b185867e772eb4066472546e8abc5983fe1e26a38c3ed4b5ecc73cb47691: Status 404 returned error can't find the container with id 2337b185867e772eb4066472546e8abc5983fe1e26a38c3ed4b5ecc73cb47691
Oct 30 21:05:11.367067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.367039    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:11.539307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.539270    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:05:11.544690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.544663    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:05:11.562515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.562472    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:05:11.629315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.629234    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9swt\" (UniqueName: \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\") pod \"authorino-56c6c9ddd5-dqt7x\" (UID: \"33177a45-e507-48e6-9eab-15d3c100ab96\") " pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:05:11.730322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.729958    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9swt\" (UniqueName: \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\") pod \"authorino-56c6c9ddd5-dqt7x\" (UID: \"33177a45-e507-48e6-9eab-15d3c100ab96\") " pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:05:11.750624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.750594    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9swt\" (UniqueName: \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\") pod \"authorino-56c6c9ddd5-dqt7x\" (UID: \"33177a45-e507-48e6-9eab-15d3c100ab96\") " pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:05:11.839177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.839138    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" event={"ID":"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659","Type":"ContainerStarted","Data":"2337b185867e772eb4066472546e8abc5983fe1e26a38c3ed4b5ecc73cb47691"}
Oct 30 21:05:11.862500 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:11.862455    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:05:12.163135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:12.163112    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:05:12.166714 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:05:12.166583    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33177a45_e507_48e6_9eab_15d3c100ab96.slice/crio-906b07f1debe57e556342d293fb9030d6346c18623963aff8d193278eb2f0ee4 WatchSource:0}: Error finding container 906b07f1debe57e556342d293fb9030d6346c18623963aff8d193278eb2f0ee4: Status 404 returned error can't find the container with id 906b07f1debe57e556342d293fb9030d6346c18623963aff8d193278eb2f0ee4
Oct 30 21:05:12.848628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:12.848587    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m" event={"ID":"aab318c8-8afe-4e74-b82e-a85e5fd1e59a","Type":"ContainerStarted","Data":"eab8d24f26b1717f670c63e53461a4336dd8166596620ef8623d998e097d5461"}
Oct 30 21:05:12.853251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:12.853217    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" event={"ID":"33177a45-e507-48e6-9eab-15d3c100ab96","Type":"ContainerStarted","Data":"906b07f1debe57e556342d293fb9030d6346c18623963aff8d193278eb2f0ee4"}
Oct 30 21:05:12.896232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:12.896169    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5649f7bc66-trj7m" podStartSLOduration=2.055813994 podStartE2EDuration="4.896147668s" totalImagesPullingTime="2.840333674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:12.894995135 +0000 UTC m=+256.306534571" watchObservedRunningTime="2026-10-30 21:05:12.896147668 +0000 UTC m=+256.307687106"
Oct 30 21:05:15.870557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:15.870513    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" event={"ID":"33177a45-e507-48e6-9eab-15d3c100ab96","Type":"ContainerStarted","Data":"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"}
Oct 30 21:05:15.872295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:15.872260    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" event={"ID":"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659","Type":"ContainerStarted","Data":"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"}
Oct 30 21:05:15.909694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:15.909641    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" podStartSLOduration=1.713390392 podStartE2EDuration="4.909625936s" totalImagesPullingTime="3.196235544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:15.908960653 +0000 UTC m=+259.320500089" watchObservedRunningTime="2026-10-30 21:05:15.909625936 +0000 UTC m=+259.321165366"
Oct 30 21:05:15.938774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:15.938740    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:15.951258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:15.951203    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" podStartSLOduration=1.903487889 podStartE2EDuration="5.951188511s" totalImagesPullingTime="4.047700622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:15.951044328 +0000 UTC m=+259.362583787" watchObservedRunningTime="2026-10-30 21:05:15.951188511 +0000 UTC m=+259.362727947"
Oct 30 21:05:16.878710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:16.878676    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg" event={"ID":"249f3602-e4df-406a-ad76-f1e605158d4d","Type":"ContainerStarted","Data":"5a99e591f71bd13977b8a37c40fd5d13b6b339496135bd6042f6c14c04fa6612"}
Oct 30 21:05:16.919384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:16.919336    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-w87qg" podStartSLOduration=2.277565454 podStartE2EDuration="7.919322513s" totalImagesPullingTime="5.641757059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:16.918557817 +0000 UTC m=+260.330097253" watchObservedRunningTime="2026-10-30 21:05:16.919322513 +0000 UTC m=+260.330861944"
Oct 30 21:05:17.882629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:17.882589    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" podUID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" containerName="authorino" containerID="cri-o://0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6" gracePeriod=30
Oct 30 21:05:18.150550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.150462    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:18.291574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.291541    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\") pod \"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659\" (UID: \"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659\") "
Oct 30 21:05:18.293705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.293676    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr" pod "0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" (UID: "0ab3a2b5-2db2-43d9-bdf8-59826bcd0659"). InnerVolumeSpecName "kube-api-access-5gnsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:18.392635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.392603    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659-kube-api-access-5gnsr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:05:18.887932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.887898    7037 generic.go:356] "Generic (PLEG): container finished" podID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" containerID="0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6" exitCode=0
Oct 30 21:05:18.888370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.887947    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-l7bhb"
Oct 30 21:05:18.888370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.887979    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" event={"ID":"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659","Type":"ContainerDied","Data":"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"}
Oct 30 21:05:18.888370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.888036    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-l7bhb" event={"ID":"0ab3a2b5-2db2-43d9-bdf8-59826bcd0659","Type":"ContainerDied","Data":"2337b185867e772eb4066472546e8abc5983fe1e26a38c3ed4b5ecc73cb47691"}
Oct 30 21:05:18.888370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.888053    7037 scope.go:122] "RemoveContainer" containerID="0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"
Oct 30 21:05:18.896115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.896098    7037 scope.go:122] "RemoveContainer" containerID="0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"
Oct 30 21:05:18.896361 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:05:18.896342    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6\": container with ID starting with 0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6 not found: ID does not exist" containerID="0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"
Oct 30 21:05:18.896421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.896372    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6"} err="failed to get container status \"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6\": rpc error: code = NotFound desc = could not find container \"0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6\": container with ID starting with 0ed09d9b1dba044914fdb66a2d96ee1d27b2daae62e16f3a7525221582c71cc6 not found: ID does not exist"
Oct 30 21:05:18.952470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.952429    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:18.964831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:18.964763    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-l7bhb"]
Oct 30 21:05:20.705210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:20.705181    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" path="/var/lib/kubelet/pods/0ab3a2b5-2db2-43d9-bdf8-59826bcd0659/volumes"
Oct 30 21:05:22.882576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.882542    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"]
Oct 30 21:05:22.882970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.882933    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" containerName="authorino"
Oct 30 21:05:22.882970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.882945    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" containerName="authorino"
Oct 30 21:05:22.883045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.883003    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab3a2b5-2db2-43d9-bdf8-59826bcd0659" containerName="authorino"
Oct 30 21:05:22.887429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.887413    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:22.915021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.914985    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:05:22.915178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.915107    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-7vmln\""
Oct 30 21:05:22.915178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.915154    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:05:22.915597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.915576    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:05:22.917096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:22.917076    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"]
Oct 30 21:05:23.033388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.033355    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-cert\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.033592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.033402    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpjd\" (UniqueName: \"kubernetes.io/projected/a1eddbb1-390c-4968-a685-69d7987fd5ca-kube-api-access-8lpjd\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.033592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.033560    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-metrics-certs\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.033592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.033585    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1eddbb1-390c-4968-a685-69d7987fd5ca-manager-config\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.134627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.134542    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-cert\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.134627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.134593    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpjd\" (UniqueName: \"kubernetes.io/projected/a1eddbb1-390c-4968-a685-69d7987fd5ca-kube-api-access-8lpjd\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.134852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.134642    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-metrics-certs\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.134852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.134661    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1eddbb1-390c-4968-a685-69d7987fd5ca-manager-config\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.135388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.135356    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1eddbb1-390c-4968-a685-69d7987fd5ca-manager-config\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.137136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.137115    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-metrics-certs\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.137203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.137151    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1eddbb1-390c-4968-a685-69d7987fd5ca-cert\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.156695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.156665    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpjd\" (UniqueName: \"kubernetes.io/projected/a1eddbb1-390c-4968-a685-69d7987fd5ca-kube-api-access-8lpjd\") pod \"jobset-controller-manager-77f9cf68b4-468q8\" (UID: \"a1eddbb1-390c-4968-a685-69d7987fd5ca\") " pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.199687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.199640    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:23.520560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.520523    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"]
Oct 30 21:05:23.521135 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:05:23.521112    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eddbb1_390c_4968_a685_69d7987fd5ca.slice/crio-14e2a2d92b9b26282391f9bc9a4ca4d25ec6f5dd2eb2d1bc41d81cbb26dbb290 WatchSource:0}: Error finding container 14e2a2d92b9b26282391f9bc9a4ca4d25ec6f5dd2eb2d1bc41d81cbb26dbb290: Status 404 returned error can't find the container with id 14e2a2d92b9b26282391f9bc9a4ca4d25ec6f5dd2eb2d1bc41d81cbb26dbb290
Oct 30 21:05:23.912722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:23.912682    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8" event={"ID":"a1eddbb1-390c-4968-a685-69d7987fd5ca","Type":"ContainerStarted","Data":"14e2a2d92b9b26282391f9bc9a4ca4d25ec6f5dd2eb2d1bc41d81cbb26dbb290"}
Oct 30 21:05:26.927791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:26.927753    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8" event={"ID":"a1eddbb1-390c-4968-a685-69d7987fd5ca","Type":"ContainerStarted","Data":"688bae1fe956519580d2f7b3feaab0a48e373fcaf5c3bf81496cf8f9ca8e6359"}
Oct 30 21:05:26.928213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:26.927848    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:05:26.974604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:26.974555    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8" podStartSLOduration=2.224915239 podStartE2EDuration="4.974541659s" totalImagesPullingTime="2.74962642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:26.973659808 +0000 UTC m=+270.385199244" watchObservedRunningTime="2026-10-30 21:05:26.974541659 +0000 UTC m=+270.386081089"
Oct 30 21:05:33.671593 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:05:33.671555    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:05:37.937463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:05:37.937378    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-77f9cf68b4-468q8"
Oct 30 21:06:03.670571 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:06:03.670533    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:06:12.655204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.655169    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-26nhx"]
Oct 30 21:06:12.658707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.658685    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.673708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.673688    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 21:06:12.686407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.686373    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:06:12.686546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.686406    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:06:12.688127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.688103    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-26nhx"]
Oct 30 21:06:12.753158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.753130    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-txqzw\""
Oct 30 21:06:12.783713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.783670    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrrb\" (UniqueName: \"kubernetes.io/projected/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-kube-api-access-lzrrb\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.783880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.783828    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-cert\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.786331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.786310    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mmhps"]
Oct 30 21:06:12.789678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.789659    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.808961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.808933    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mmhps"]
Oct 30 21:06:12.873928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.873897    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:06:12.874124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.874103    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9mdqv\""
Oct 30 21:06:12.874167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.874103    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:06:12.885267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.885238    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrrb\" (UniqueName: \"kubernetes.io/projected/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-kube-api-access-lzrrb\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.885411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.885288    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfbn\" (UniqueName: \"kubernetes.io/projected/6b818c11-e7eb-495d-b7f7-949553059316-kube-api-access-9pfbn\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.885411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.885328    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-cert\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.885539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.885422    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-metrics-certs\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.885539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.885464    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-cert\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.887757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.887735    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-cert\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.907906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.907843    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrrb\" (UniqueName: \"kubernetes.io/projected/cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b-kube-api-access-lzrrb\") pod \"kserve-controller-manager-776cc66d47-26nhx\" (UID: \"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.971455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.971416    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:12.986911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.986861    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-metrics-certs\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.987050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.986940    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-cert\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.987050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.987027    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfbn\" (UniqueName: \"kubernetes.io/projected/6b818c11-e7eb-495d-b7f7-949553059316-kube-api-access-9pfbn\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.989562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.989475    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-metrics-certs\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:12.989659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:12.989547    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b818c11-e7eb-495d-b7f7-949553059316-cert\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:13.007240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:13.007209    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfbn\" (UniqueName: \"kubernetes.io/projected/6b818c11-e7eb-495d-b7f7-949553059316-kube-api-access-9pfbn\") pod \"odh-model-controller-769875db56-mmhps\" (UID: \"6b818c11-e7eb-495d-b7f7-949553059316\") " pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:13.101690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:13.101654    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:13.265035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:13.265010    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-26nhx"]
Oct 30 21:06:13.265785 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:06:13.265753    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6a26a4_4a55_41e5_9b59_1cc6fb32fc7b.slice/crio-18f40ad7c45e79f276e0da7845a72dbb0055b04ff4c2291fcd693ce1bda48442 WatchSource:0}: Error finding container 18f40ad7c45e79f276e0da7845a72dbb0055b04ff4c2291fcd693ce1bda48442: Status 404 returned error can't find the container with id 18f40ad7c45e79f276e0da7845a72dbb0055b04ff4c2291fcd693ce1bda48442
Oct 30 21:06:13.396628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:13.396594    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mmhps"]
Oct 30 21:06:13.396968 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:06:13.396947    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b818c11_e7eb_495d_b7f7_949553059316.slice/crio-dcc4f8235e549f1eb982fb06bf1143ca429dc9712f201d7820af2019d2e9721e WatchSource:0}: Error finding container dcc4f8235e549f1eb982fb06bf1143ca429dc9712f201d7820af2019d2e9721e: Status 404 returned error can't find the container with id dcc4f8235e549f1eb982fb06bf1143ca429dc9712f201d7820af2019d2e9721e
Oct 30 21:06:14.160389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:14.160348    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mmhps" event={"ID":"6b818c11-e7eb-495d-b7f7-949553059316","Type":"ContainerStarted","Data":"dcc4f8235e549f1eb982fb06bf1143ca429dc9712f201d7820af2019d2e9721e"}
Oct 30 21:06:14.162228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:14.162195    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx" event={"ID":"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b","Type":"ContainerStarted","Data":"18f40ad7c45e79f276e0da7845a72dbb0055b04ff4c2291fcd693ce1bda48442"}
Oct 30 21:06:16.174555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:16.174523    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx" event={"ID":"cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b","Type":"ContainerStarted","Data":"e405d592535bd7f7c36d817dbb4707cb5602438b80cb57fc55a17bebebf055aa"}
Oct 30 21:06:16.174934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:16.174651    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:16.226820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:16.226756    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx" podStartSLOduration=1.8808576380000002 podStartE2EDuration="4.226735589s" totalImagesPullingTime="2.345877951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:16.225264054 +0000 UTC m=+319.636803498" watchObservedRunningTime="2026-10-30 21:06:16.226735589 +0000 UTC m=+319.638275031"
Oct 30 21:06:17.182227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:17.182186    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mmhps" event={"ID":"6b818c11-e7eb-495d-b7f7-949553059316","Type":"ContainerStarted","Data":"ab414e7613a0368dc3e980ea1e2e9d6f9e832554b4e2a38d5639ab06f028074d"}
Oct 30 21:06:17.182667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:17.182363    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:28.189459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:28.189427    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mmhps"
Oct 30 21:06:28.234174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:28.234111    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mmhps" podStartSLOduration=13.510744777 podStartE2EDuration="16.234094433s" totalImagesPullingTime="2.723349656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:17.223935676 +0000 UTC m=+320.635475113" watchObservedRunningTime="2026-10-30 21:06:28.234094433 +0000 UTC m=+331.645633912"
Oct 30 21:06:33.671958 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:06:33.671921    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:06:47.187320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:47.187286    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-26nhx"
Oct 30 21:06:56.725026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:06:56.724995    7037 scope.go:122] "RemoveContainer" containerID="d41f6145e7fa093d953c3c92d6af819e0d462f2067d2e4d5ff35972bcfbc2caf"
Oct 30 21:07:03.671207 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:07:03.671169    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:07:15.769081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.769037    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wk726"]
Oct 30 21:07:15.771633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.771609    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:15.797305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.797281    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wk726"]
Oct 30 21:07:15.858956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.858919    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb85d\" (UniqueName: \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\") pod \"authorino-6f6944bcdd-wk726\" (UID: \"30e25eb1-97e6-4269-8a66-609f84162d66\") " pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:15.949325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.949290    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wk726"]
Oct 30 21:07:15.949564 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:07:15.949545    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mb85d], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-wk726" podUID="30e25eb1-97e6-4269-8a66-609f84162d66"
Oct 30 21:07:15.959673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.959644    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb85d\" (UniqueName: \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\") pod \"authorino-6f6944bcdd-wk726\" (UID: \"30e25eb1-97e6-4269-8a66-609f84162d66\") " pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:15.974595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.974565    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6wn7m"]
Oct 30 21:07:15.981801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.981767    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:15.983025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.982999    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb85d\" (UniqueName: \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\") pod \"authorino-6f6944bcdd-wk726\" (UID: \"30e25eb1-97e6-4269-8a66-609f84162d66\") " pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:15.995023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.994996    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6wn7m"]
Oct 30 21:07:15.997177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:15.997126    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:07:16.060924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.060846    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.061060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.060978    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9brg\" (UniqueName: \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.136507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.136469    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6wn7m"]
Oct 30 21:07:16.136714 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:07:16.136696    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j9brg tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-6wn7m" podUID="2940f469-3cf7-44e0-816a-2465e1f26dda"
Oct 30 21:07:16.162240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.162206    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.162405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.162378    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9brg\" (UniqueName: \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.164813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.164786    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.168238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.168212    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:07:16.170646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.170630    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.190754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.190728    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9brg\" (UniqueName: \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\") pod \"authorino-6889cbd8c4-6wn7m\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") " pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.192070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.192045    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:07:16.263646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.263615    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2n8\" (UniqueName: \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.263804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.263676    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.364226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.364183    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2n8\" (UniqueName: \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.364417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.364337    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.366709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.366688    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.383054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.383027    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2n8\" (UniqueName: \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\") pod \"authorino-659f7c59b-lpwwl\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") " pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.448734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.448686    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.448923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.448687    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:16.454972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.454950    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:16.458093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.458074    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:16.483470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.483424    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:07:16.566788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.566760    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\" (UniqueName: \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\") pod \"30e25eb1-97e6-4269-8a66-609f84162d66\" (UID: \"30e25eb1-97e6-4269-8a66-609f84162d66\") "
Oct 30 21:07:16.566928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.566794    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\" (UniqueName: \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\") pod \"2940f469-3cf7-44e0-816a-2465e1f26dda\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") "
Oct 30 21:07:16.566928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.566868    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\" (UniqueName: \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\") pod \"2940f469-3cf7-44e0-816a-2465e1f26dda\" (UID: \"2940f469-3cf7-44e0-816a-2465e1f26dda\") "
Oct 30 21:07:16.568883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.568856    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert" pod "2940f469-3cf7-44e0-816a-2465e1f26dda" (UID: "2940f469-3cf7-44e0-816a-2465e1f26dda"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:16.568883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.568871    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d" pod "30e25eb1-97e6-4269-8a66-609f84162d66" (UID: "30e25eb1-97e6-4269-8a66-609f84162d66"). InnerVolumeSpecName "kube-api-access-mb85d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:16.569027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.568884    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg" pod "2940f469-3cf7-44e0-816a-2465e1f26dda" (UID: "2940f469-3cf7-44e0-816a-2465e1f26dda"). InnerVolumeSpecName "kube-api-access-j9brg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:16.668407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.668323    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb85d\" (UniqueName: \"kubernetes.io/projected/30e25eb1-97e6-4269-8a66-609f84162d66-kube-api-access-mb85d\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:07:16.668407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.668354    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9brg\" (UniqueName: \"kubernetes.io/projected/2940f469-3cf7-44e0-816a-2465e1f26dda-kube-api-access-j9brg\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:07:16.668407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.668363    7037 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2940f469-3cf7-44e0-816a-2465e1f26dda-tls-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:07:16.773321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:16.773294    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:07:16.774288 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:07:16.774261    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6de5d9c7_fc22_43c5_b0cd_e3e643aef99f.slice/crio-acfc29296aed2187a4cd84c474e4f09adab961cabe0379f7732cd94d5410fde3 WatchSource:0}: Error finding container acfc29296aed2187a4cd84c474e4f09adab961cabe0379f7732cd94d5410fde3: Status 404 returned error can't find the container with id acfc29296aed2187a4cd84c474e4f09adab961cabe0379f7732cd94d5410fde3
Oct 30 21:07:17.455541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.455464    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659f7c59b-lpwwl" event={"ID":"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f","Type":"ContainerStarted","Data":"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"}
Oct 30 21:07:17.455541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.455531    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wk726"
Oct 30 21:07:17.455783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.455556    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659f7c59b-lpwwl" event={"ID":"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f","Type":"ContainerStarted","Data":"acfc29296aed2187a4cd84c474e4f09adab961cabe0379f7732cd94d5410fde3"}
Oct 30 21:07:17.455914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.455890    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6wn7m"
Oct 30 21:07:17.509070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.508668    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-659f7c59b-lpwwl" podStartSLOduration=1.138127511 podStartE2EDuration="1.508649367s" totalImagesPullingTime="370.521856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:17.50814093 +0000 UTC m=+380.919680368" watchObservedRunningTime="2026-10-30 21:07:17.508649367 +0000 UTC m=+380.920188805"
Oct 30 21:07:17.538612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.538572    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:07:17.538835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.538812    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" podUID="33177a45-e507-48e6-9eab-15d3c100ab96" containerName="authorino" containerID="cri-o://dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663" gracePeriod=30
Oct 30 21:07:17.579599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.579560    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wk726"]
Oct 30 21:07:17.587461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.587431    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wk726"]
Oct 30 21:07:17.645767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.645677    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6wn7m"]
Oct 30 21:07:17.648751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.648721    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6wn7m"]
Oct 30 21:07:17.783295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.783270    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:07:17.879724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.879638    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\" (UniqueName: \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\") pod \"33177a45-e507-48e6-9eab-15d3c100ab96\" (UID: \"33177a45-e507-48e6-9eab-15d3c100ab96\") "
Oct 30 21:07:17.881720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.881682    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt" pod "33177a45-e507-48e6-9eab-15d3c100ab96" (UID: "33177a45-e507-48e6-9eab-15d3c100ab96"). InnerVolumeSpecName "kube-api-access-q9swt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:17.980300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:17.980264    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9swt\" (UniqueName: \"kubernetes.io/projected/33177a45-e507-48e6-9eab-15d3c100ab96-kube-api-access-q9swt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:07:18.462301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.462260    7037 generic.go:356] "Generic (PLEG): container finished" podID="33177a45-e507-48e6-9eab-15d3c100ab96" containerID="dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663" exitCode=0
Oct 30 21:07:18.462301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.462303    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x"
Oct 30 21:07:18.462692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.462339    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" event={"ID":"33177a45-e507-48e6-9eab-15d3c100ab96","Type":"ContainerDied","Data":"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"}
Oct 30 21:07:18.462692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.462398    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dqt7x" event={"ID":"33177a45-e507-48e6-9eab-15d3c100ab96","Type":"ContainerDied","Data":"906b07f1debe57e556342d293fb9030d6346c18623963aff8d193278eb2f0ee4"}
Oct 30 21:07:18.462692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.462424    7037 scope.go:122] "RemoveContainer" containerID="dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"
Oct 30 21:07:18.473065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.473043    7037 scope.go:122] "RemoveContainer" containerID="dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"
Oct 30 21:07:18.473362 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:07:18.473343    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663\": container with ID starting with dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663 not found: ID does not exist" containerID="dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"
Oct 30 21:07:18.473407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.473374    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663"} err="failed to get container status \"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663\": rpc error: code = NotFound desc = could not find container \"dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663\": container with ID starting with dddeb1505ff8378c2b4b982b92af2dadd844579e29400e4d65f20b223e83c663 not found: ID does not exist"
Oct 30 21:07:18.511766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.511738    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:07:18.523972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.523942    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dqt7x"]
Oct 30 21:07:18.706466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.706426    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2940f469-3cf7-44e0-816a-2465e1f26dda" path="/var/lib/kubelet/pods/2940f469-3cf7-44e0-816a-2465e1f26dda/volumes"
Oct 30 21:07:18.706846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.706827    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e25eb1-97e6-4269-8a66-609f84162d66" path="/var/lib/kubelet/pods/30e25eb1-97e6-4269-8a66-609f84162d66/volumes"
Oct 30 21:07:18.707171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:07:18.707150    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33177a45-e507-48e6-9eab-15d3c100ab96" path="/var/lib/kubelet/pods/33177a45-e507-48e6-9eab-15d3c100ab96/volumes"
Oct 30 21:07:33.671059 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:07:33.671026    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:08:03.670425 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:08:03.670381    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:08:21.188955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.188914    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"]
Oct 30 21:08:21.189415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.189334    7037 cpu_manager.go:414] "containerMap: removing container" podUID="33177a45-e507-48e6-9eab-15d3c100ab96" containerName="authorino"
Oct 30 21:08:21.189415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.189344    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33177a45-e507-48e6-9eab-15d3c100ab96" containerName="authorino"
Oct 30 21:08:21.189415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.189404    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="33177a45-e507-48e6-9eab-15d3c100ab96" containerName="authorino"
Oct 30 21:08:21.191535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.191479    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.218775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.218743    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 21:08:21.218775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.218773    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:08:21.218994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.218804    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sd4tj\""
Oct 30 21:08:21.218994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.218887    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:08:21.220936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.220914    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"]
Oct 30 21:08:21.356451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356403    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.356451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356448    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.356749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zs8w\" (UniqueName: \"kubernetes.io/projected/c655b016-b634-484e-a598-4611ee2a6be6-kube-api-access-6zs8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.356749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356643    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.356749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356665    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c655b016-b634-484e-a598-4611ee2a6be6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.356749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.356736    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458132    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458176    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458300    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs8w\" (UniqueName: \"kubernetes.io/projected/c655b016-b634-484e-a598-4611ee2a6be6-kube-api-access-6zs8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458455    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458486    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c655b016-b634-484e-a598-4611ee2a6be6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458561    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458620    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.458935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458908    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.459026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.458944    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.460523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.460484    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c655b016-b634-484e-a598-4611ee2a6be6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.460986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.460965    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c655b016-b634-484e-a598-4611ee2a6be6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.477711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.477684    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs8w\" (UniqueName: \"kubernetes.io/projected/c655b016-b634-484e-a598-4611ee2a6be6-kube-api-access-6zs8w\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nhzxz\" (UID: \"c655b016-b634-484e-a598-4611ee2a6be6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.505622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.505590    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:21.807970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:21.807937    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"]
Oct 30 21:08:21.808780 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:21.808756    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc655b016_b634_484e_a598_4611ee2a6be6.slice/crio-4385db383b86551f3b4446c2114ba4899f0039178c9153b54b0930580084479a WatchSource:0}: Error finding container 4385db383b86551f3b4446c2114ba4899f0039178c9153b54b0930580084479a: Status 404 returned error can't find the container with id 4385db383b86551f3b4446c2114ba4899f0039178c9153b54b0930580084479a
Oct 30 21:08:22.388220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.388186    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"]
Oct 30 21:08:22.393414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.393382    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.425762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.425704    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:08:22.427653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.427628    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"]
Oct 30 21:08:22.469823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.469791    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.469995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.469832    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.469995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.469977    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.470137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.470011    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.470137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.470040    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd87b\" (UniqueName: \"kubernetes.io/projected/5e30ce33-adeb-4935-91db-42bf140e78fe-kube-api-access-rd87b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.470243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.470134    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e30ce33-adeb-4935-91db-42bf140e78fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.571806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.571762    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.571823    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.571908    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.571937    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.571970    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd87b\" (UniqueName: \"kubernetes.io/projected/5e30ce33-adeb-4935-91db-42bf140e78fe-kube-api-access-rd87b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.572020    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e30ce33-adeb-4935-91db-42bf140e78fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.572322    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572851 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.572513    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.572851 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.572815    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.574276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.574254    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e30ce33-adeb-4935-91db-42bf140e78fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.574759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.574737    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e30ce33-adeb-4935-91db-42bf140e78fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.592209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.592172    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd87b\" (UniqueName: \"kubernetes.io/projected/5e30ce33-adeb-4935-91db-42bf140e78fe-kube-api-access-rd87b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2\" (UID: \"5e30ce33-adeb-4935-91db-42bf140e78fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.712224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.712146    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:22.753252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:22.753180    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz" event={"ID":"c655b016-b634-484e-a598-4611ee2a6be6","Type":"ContainerStarted","Data":"4385db383b86551f3b4446c2114ba4899f0039178c9153b54b0930580084479a"}
Oct 30 21:08:23.012214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.012183    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"]
Oct 30 21:08:23.012575 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:23.012542    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e30ce33_adeb_4935_91db_42bf140e78fe.slice/crio-adecf6df7dd7cbcae9a12309f0e77ce2c9955c85526e574c71eafd838b7ccf9c WatchSource:0}: Error finding container adecf6df7dd7cbcae9a12309f0e77ce2c9955c85526e574c71eafd838b7ccf9c: Status 404 returned error can't find the container with id adecf6df7dd7cbcae9a12309f0e77ce2c9955c85526e574c71eafd838b7ccf9c
Oct 30 21:08:23.760389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.760344    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz" event={"ID":"c655b016-b634-484e-a598-4611ee2a6be6","Type":"ContainerStarted","Data":"f58f39ddd8efdfc0e23b94ca98cb91a54a9ce9e6fa78c82a744dbec3ae472b43"}
Oct 30 21:08:23.762060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.762027    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2" event={"ID":"5e30ce33-adeb-4935-91db-42bf140e78fe","Type":"ContainerStarted","Data":"e8f33aefd31d610c6d0fd71ac3960f4cd0f260dba6d4be3d344e21b20b3f420a"}
Oct 30 21:08:23.762176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.762088    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2" event={"ID":"5e30ce33-adeb-4935-91db-42bf140e78fe","Type":"ContainerStarted","Data":"adecf6df7dd7cbcae9a12309f0e77ce2c9955c85526e574c71eafd838b7ccf9c"}
Oct 30 21:08:23.801451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.801397    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz" podStartSLOduration=1.721464445 podStartE2EDuration="2.801384947s" totalImagesPullingTime="1.079920502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:23.800380927 +0000 UTC m=+447.211920363" watchObservedRunningTime="2026-10-30 21:08:23.801384947 +0000 UTC m=+447.212924381"
Oct 30 21:08:23.840575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:23.840512    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2" podStartSLOduration=1.650754054 podStartE2EDuration="1.840478259s" totalImagesPullingTime="189.724205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:23.839286466 +0000 UTC m=+447.250825904" watchObservedRunningTime="2026-10-30 21:08:23.840478259 +0000 UTC m=+447.252017901"
Oct 30 21:08:24.505776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:24.505736    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:24.712605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:24.712573    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:26.276004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.275969    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"]
Oct 30 21:08:26.278988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.278962    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.298636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.298594    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"]
Oct 30 21:08:26.309073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.309051    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:08:26.406327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406296    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf84\" (UniqueName: \"kubernetes.io/projected/2ba11244-cf05-4477-af93-89045c8c4bdd-kube-api-access-nzf84\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.406484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406349    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.406484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406369    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.406484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406468    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.406711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406529    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.406711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.406647    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba11244-cf05-4477-af93-89045c8c4bdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.507689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.507652    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba11244-cf05-4477-af93-89045c8c4bdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.507883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.507700    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf84\" (UniqueName: \"kubernetes.io/projected/2ba11244-cf05-4477-af93-89045c8c4bdd-kube-api-access-nzf84\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.507883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.507874    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.507903    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.507938    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.508043    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.508256    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.508319    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.508423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.508392    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.510061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.510039    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ba11244-cf05-4477-af93-89045c8c4bdd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.510323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.510305    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ba11244-cf05-4477-af93-89045c8c4bdd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.526813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.526759    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf84\" (UniqueName: \"kubernetes.io/projected/2ba11244-cf05-4477-af93-89045c8c4bdd-kube-api-access-nzf84\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb\" (UID: \"2ba11244-cf05-4477-af93-89045c8c4bdd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.592793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.592772    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:26.882970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:26.882938    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"]
Oct 30 21:08:26.883450 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:26.883427    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba11244_cf05_4477_af93_89045c8c4bdd.slice/crio-8d534badd9e47052acd413f355d5ccc47e89849357459b35bfaea0e41771fef2 WatchSource:0}: Error finding container 8d534badd9e47052acd413f355d5ccc47e89849357459b35bfaea0e41771fef2: Status 404 returned error can't find the container with id 8d534badd9e47052acd413f355d5ccc47e89849357459b35bfaea0e41771fef2
Oct 30 21:08:27.782256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:27.782216    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb" event={"ID":"2ba11244-cf05-4477-af93-89045c8c4bdd","Type":"ContainerStarted","Data":"0bde77b5e5bed942131b38fe0396da79d9cb0a57bb0e7ff0fbc6b0576473c6b4"}
Oct 30 21:08:27.782673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:27.782278    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb" event={"ID":"2ba11244-cf05-4477-af93-89045c8c4bdd","Type":"ContainerStarted","Data":"8d534badd9e47052acd413f355d5ccc47e89849357459b35bfaea0e41771fef2"}
Oct 30 21:08:27.821975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:27.821925    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb" podStartSLOduration=1.609956202 podStartE2EDuration="1.821912966s" totalImagesPullingTime="211.956764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:27.821052332 +0000 UTC m=+451.232591768" watchObservedRunningTime="2026-10-30 21:08:27.821912966 +0000 UTC m=+451.233452402"
Oct 30 21:08:28.593144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:28.593111    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:31.505959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:31.505914    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:31.518769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:31.518742    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:31.818030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:31.818002    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nhzxz"
Oct 30 21:08:32.713026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:32.712989    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:32.726020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:32.725987    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:32.828481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:32.828449    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2"
Oct 30 21:08:33.670261 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:08:33.670216    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:08:36.593582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:36.593471    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:36.605830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:36.605803    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:36.845701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:36.845618    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb"
Oct 30 21:08:43.772688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.772651    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-797c9987-ddv6q"]
Oct 30 21:08:43.775020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.775003    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.794852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.794825    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797c9987-ddv6q"]
Oct 30 21:08:43.870449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.870411    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea5787a9-057e-48c5-9ca9-7c3f3400149c-tls-cert\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.870652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.870515    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjnl\" (UniqueName: \"kubernetes.io/projected/ea5787a9-057e-48c5-9ca9-7c3f3400149c-kube-api-access-8fjnl\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.971055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.971020    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea5787a9-057e-48c5-9ca9-7c3f3400149c-tls-cert\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.971214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.971082    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjnl\" (UniqueName: \"kubernetes.io/projected/ea5787a9-057e-48c5-9ca9-7c3f3400149c-kube-api-access-8fjnl\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.973372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.973351    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea5787a9-057e-48c5-9ca9-7c3f3400149c-tls-cert\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:43.989713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:43.989687    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjnl\" (UniqueName: \"kubernetes.io/projected/ea5787a9-057e-48c5-9ca9-7c3f3400149c-kube-api-access-8fjnl\") pod \"authorino-797c9987-ddv6q\" (UID: \"ea5787a9-057e-48c5-9ca9-7c3f3400149c\") " pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:44.085799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:44.085764    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797c9987-ddv6q"
Oct 30 21:08:44.380955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:44.380923    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797c9987-ddv6q"]
Oct 30 21:08:44.381519 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:44.381475    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea5787a9_057e_48c5_9ca9_7c3f3400149c.slice/crio-c62e24f9aec2949d4fc302c466d53085bfe06b4d3510d965d3d459b21d6a019c WatchSource:0}: Error finding container c62e24f9aec2949d4fc302c466d53085bfe06b4d3510d965d3d459b21d6a019c: Status 404 returned error can't find the container with id c62e24f9aec2949d4fc302c466d53085bfe06b4d3510d965d3d459b21d6a019c
Oct 30 21:08:44.873507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:44.873457    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797c9987-ddv6q" event={"ID":"ea5787a9-057e-48c5-9ca9-7c3f3400149c","Type":"ContainerStarted","Data":"c62e24f9aec2949d4fc302c466d53085bfe06b4d3510d965d3d459b21d6a019c"}
Oct 30 21:08:45.884534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:45.884474    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797c9987-ddv6q" event={"ID":"ea5787a9-057e-48c5-9ca9-7c3f3400149c","Type":"ContainerStarted","Data":"f734a5b612337d7ce8350f4045d4d0acb946d723386af9cda2e9faa8f6c738e6"}
Oct 30 21:08:45.924116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:45.924058    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-797c9987-ddv6q" podStartSLOduration=2.3883169410000002 podStartE2EDuration="2.924045749s" totalImagesPullingTime="535.728808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:45.923549811 +0000 UTC m=+469.335089245" watchObservedRunningTime="2026-10-30 21:08:45.924045749 +0000 UTC m=+469.335585183"
Oct 30 21:08:45.950464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:45.950410    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:08:45.950647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:45.950619    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-659f7c59b-lpwwl" podUID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" containerName="authorino" containerID="cri-o://dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f" gracePeriod=30
Oct 30 21:08:46.197803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.197776    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:08:46.292089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.292056    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\" (UniqueName: \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\") pod \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") "
Oct 30 21:08:46.292245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.292180    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\" (UniqueName: \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\") pod \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\" (UID: \"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f\") "
Oct 30 21:08:46.294211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.294179    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8" pod "6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" (UID: "6de5d9c7-fc22-43c5-b0cd-e3e643aef99f"). InnerVolumeSpecName "kube-api-access-wk2n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:46.302303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.302278    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert" pod "6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" (UID: "6de5d9c7-fc22-43c5-b0cd-e3e643aef99f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:46.392837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.392805    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2n8\" (UniqueName: \"kubernetes.io/projected/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-kube-api-access-wk2n8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:08:46.392837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.392833    7037 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f-tls-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:08:46.522254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.522167    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:08:46.522884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.522862    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" containerName="authorino"
Oct 30 21:08:46.522964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.522888    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" containerName="authorino"
Oct 30 21:08:46.523045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.523032    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" containerName="authorino"
Oct 30 21:08:46.525666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.525644    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.543806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.543786    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:08:46.548065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.548035    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:08:46.548163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.548133    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-s4mgb\""
Oct 30 21:08:46.551522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.549629    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:08:46.595166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.595129    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.595331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.595309    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbdw\" (UniqueName: \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.696614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.696577    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.696792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.696659    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbdw\" (UniqueName: \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.699366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.699335    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.724901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.724870    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbdw\" (UniqueName: \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\") pod \"ai-gateway-controller-655fc879db-6gcwj\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") " pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.837437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.837403    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:46.899127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.899096    7037 generic.go:356] "Generic (PLEG): container finished" podID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" containerID="dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f" exitCode=0
Oct 30 21:08:46.899661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.899638    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659f7c59b-lpwwl"
Oct 30 21:08:46.899957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.899928    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659f7c59b-lpwwl" event={"ID":"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f","Type":"ContainerDied","Data":"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"}
Oct 30 21:08:46.900024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.899975    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659f7c59b-lpwwl" event={"ID":"6de5d9c7-fc22-43c5-b0cd-e3e643aef99f","Type":"ContainerDied","Data":"acfc29296aed2187a4cd84c474e4f09adab961cabe0379f7732cd94d5410fde3"}
Oct 30 21:08:46.900024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.899997    7037 scope.go:122] "RemoveContainer" containerID="dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"
Oct 30 21:08:46.912911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.912885    7037 scope.go:122] "RemoveContainer" containerID="dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"
Oct 30 21:08:46.913449 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:08:46.913416    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f\": container with ID starting with dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f not found: ID does not exist" containerID="dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"
Oct 30 21:08:46.913605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.913462    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f"} err="failed to get container status \"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f\": rpc error: code = NotFound desc = could not find container \"dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f\": container with ID starting with dfe0f7c1b398e9ede306fa863377da2e2b2a68e032acb992b6d4a2e0a0d01d6f not found: ID does not exist"
Oct 30 21:08:46.952572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.952423    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:08:46.965244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:46.965211    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-659f7c59b-lpwwl"]
Oct 30 21:08:47.134128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.134093    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:08:47.135196 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:47.135163    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a65e60_e7c3_4485_9bd6_171a7226be7a.slice/crio-e1f17345d7844c849ed8e1b4178b39fae3c41514c980a6e980f2b3d494ba42c2 WatchSource:0}: Error finding container e1f17345d7844c849ed8e1b4178b39fae3c41514c980a6e980f2b3d494ba42c2: Status 404 returned error can't find the container with id e1f17345d7844c849ed8e1b4178b39fae3c41514c980a6e980f2b3d494ba42c2
Oct 30 21:08:47.376048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.375998    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:08:47.379804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.379779    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:08:47.379945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.379905    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.382299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.382242    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.401787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.401517    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:08:47.403849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.403410    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404176    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404232    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb86z\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404264    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404353    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404393    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404422    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sls9z\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404467    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404513    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404549    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404594    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404620    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404667    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404698    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404739    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404765    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404794    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.404888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.404828    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.405821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.405800    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:08:47.411718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.411696    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:08:47.486088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.486049    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w59m6\""
Oct 30 21:08:47.505373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505338    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505383    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.505526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505419    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sls9z\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505454    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505477    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505636    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505672    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505701    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505736    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505767    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.505799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505795    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505820    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505850    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505883    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505887    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505948    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505951    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.505985    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506010    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb86z\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506042    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506353    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506576    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506594    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506872    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.506963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506887    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.506963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.506918    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.507314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.507292    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.507665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.507622    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.509360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.509333    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.509466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.509387    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.509592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.509468    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.509875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.509852    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.525217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.525192    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sls9z\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.525608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.525584    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.525751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.525727    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-lkltg\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.525917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.525903    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb86z\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-p8tnk\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.697903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.697804    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:47.712148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.712119    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:47.908136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:47.908091    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" event={"ID":"c4a65e60-e7c3-4485-9bd6-171a7226be7a","Type":"ContainerStarted","Data":"e1f17345d7844c849ed8e1b4178b39fae3c41514c980a6e980f2b3d494ba42c2"}
Oct 30 21:08:48.019538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:48.019451    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:08:48.036545 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:48.036505    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae3eda8_0abe_45c3_ae7e_59bd41848798.slice/crio-98ed41c340b90bd0ced46ebbe422224062b05d4aeba8e346b720636b0481db2d WatchSource:0}: Error finding container 98ed41c340b90bd0ced46ebbe422224062b05d4aeba8e346b720636b0481db2d: Status 404 returned error can't find the container with id 98ed41c340b90bd0ced46ebbe422224062b05d4aeba8e346b720636b0481db2d
Oct 30 21:08:48.053784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:48.053681    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:08:48.057680 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:08:48.057641    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fb5e30_cc99_44c6_a61f_7c0d04f72aaf.slice/crio-3bf839cc381dbd9f149e3bd7bd0e10d298fb7dfee4a466a9d0bed2003ac8f0a5 WatchSource:0}: Error finding container 3bf839cc381dbd9f149e3bd7bd0e10d298fb7dfee4a466a9d0bed2003ac8f0a5: Status 404 returned error can't find the container with id 3bf839cc381dbd9f149e3bd7bd0e10d298fb7dfee4a466a9d0bed2003ac8f0a5
Oct 30 21:08:48.714202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:48.714163    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de5d9c7-fc22-43c5-b0cd-e3e643aef99f" path="/var/lib/kubelet/pods/6de5d9c7-fc22-43c5-b0cd-e3e643aef99f/volumes"
Oct 30 21:08:48.922900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:48.922854    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" event={"ID":"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf","Type":"ContainerStarted","Data":"3bf839cc381dbd9f149e3bd7bd0e10d298fb7dfee4a466a9d0bed2003ac8f0a5"}
Oct 30 21:08:48.925939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:48.925900    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" event={"ID":"4ae3eda8-0abe-45c3-ae7e-59bd41848798","Type":"ContainerStarted","Data":"98ed41c340b90bd0ced46ebbe422224062b05d4aeba8e346b720636b0481db2d"}
Oct 30 21:08:51.795709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.795672    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.796026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.795753    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.796026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.795785    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.805094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.805063    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.805196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.805127    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.805196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.805160    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:08:51.954584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.954477    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" event={"ID":"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf","Type":"ContainerStarted","Data":"bd344a53c5d667c91322e8c59240ae4597a4f6970f4ff6b50c315efd81c926d4"}
Oct 30 21:08:51.956924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.956890    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" event={"ID":"4ae3eda8-0abe-45c3-ae7e-59bd41848798","Type":"ContainerStarted","Data":"e4a5c5ab00442ead7ac53d3c7c740d4af954919ea5740c2b044a915a56e5d1e7"}
Oct 30 21:08:51.958888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.958858    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" event={"ID":"c4a65e60-e7c3-4485-9bd6-171a7226be7a","Type":"ContainerStarted","Data":"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"}
Oct 30 21:08:51.959375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:51.959358    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:08:52.021918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.021865    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podStartSLOduration=1.288801331 podStartE2EDuration="5.021851079s" totalImagesPullingTime="3.733049748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.009190533 +0000 UTC m=+475.420729973" watchObservedRunningTime="2026-10-30 21:08:52.021851079 +0000 UTC m=+475.433390516"
Oct 30 21:08:52.047791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.047692    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" podStartSLOduration=2.155689599 podStartE2EDuration="6.047677819s" totalImagesPullingTime="3.89198822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.046838958 +0000 UTC m=+475.458378397" watchObservedRunningTime="2026-10-30 21:08:52.047677819 +0000 UTC m=+475.459217255"
Oct 30 21:08:52.089254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.089186    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podStartSLOduration=1.323815367 podStartE2EDuration="5.089172714s" totalImagesPullingTime="3.765357347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:52.087481918 +0000 UTC m=+475.499021355" watchObservedRunningTime="2026-10-30 21:08:52.089172714 +0000 UTC m=+475.500712151"
Oct 30 21:08:52.698838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.698801    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:08:52.700434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.700389    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:52.700583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.700459    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:52.712861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.712834    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:08:52.713956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.713936    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:52.714044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:52.713977    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:53.698302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:53.698256    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:53.698935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:53.698324    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:53.713629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:53.713602    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:53.713751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:53.713648    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:54.698823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:54.698788    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:54.700604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:54.700565    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:54.713039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:54.713012    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:54.713151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:54.713066    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:55.698700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:55.698667    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:55.698943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:55.698922    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:55.713239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:55.713209    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:55.713354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:55.713260    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:56.698308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:56.698268    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:56.698471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:56.698319    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:56.712887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:56.712858    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:56.713226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:56.712901    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:57.699060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:57.699024    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:57.699290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:57.699074    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:57.713005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:57.712979    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:57.713318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:57.713021    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:08:58.698974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:58.698928    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:08:58.699150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:58.699004    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:08:58.712972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:58.712940    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:08:58.713118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:08:58.712994    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:09:00.698800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:00.698742    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:09:00.699243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:00.698834    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:00.713645 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:00.713569    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:09:00.713827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:00.713669    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:01.739726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:01.739677    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-lkltg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:09:01.740286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:01.739785    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:01.754675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:01.754625    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-p8tnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:09:01.754842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:01.754691    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:02.321909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.321879    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:09:02.322196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.322174    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:09:02.322917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.322895    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:09:02.635604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.635484    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:09:02.636032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.635969    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:09:02.637071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:02.637047    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:09:03.672435 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:09:03.672396    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:09:03.982085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:03.982012    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:09:04.308964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.308884    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-9b9d764bc-9m9td"]
Oct 30 21:09:04.314820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.314798    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.340837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.340803    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9b9d764bc-9m9td"]
Oct 30 21:09:04.368753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.368724    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3837a8c5-b627-4a55-b2bc-7cfcede7214c-metrics-certs\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.368925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.368762    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zjl\" (UniqueName: \"kubernetes.io/projected/3837a8c5-b627-4a55-b2bc-7cfcede7214c-kube-api-access-f5zjl\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.469655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.469613    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3837a8c5-b627-4a55-b2bc-7cfcede7214c-metrics-certs\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.469830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.469660    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zjl\" (UniqueName: \"kubernetes.io/projected/3837a8c5-b627-4a55-b2bc-7cfcede7214c-kube-api-access-f5zjl\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.472088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.472056    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3837a8c5-b627-4a55-b2bc-7cfcede7214c-metrics-certs\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.489819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.489790    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zjl\" (UniqueName: \"kubernetes.io/projected/3837a8c5-b627-4a55-b2bc-7cfcede7214c-kube-api-access-f5zjl\") pod \"ai-gateway-controller-9b9d764bc-9m9td\" (UID: \"3837a8c5-b627-4a55-b2bc-7cfcede7214c\") " pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.628092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.628057    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:04.918904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.918867    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9b9d764bc-9m9td"]
Oct 30 21:09:04.919547 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:04.919518    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3837a8c5_b627_4a55_b2bc_7cfcede7214c.slice/crio-63cb668f6be31deb92c0bf5002b88716c90346286b720c6eb73c2b6286f93161 WatchSource:0}: Error finding container 63cb668f6be31deb92c0bf5002b88716c90346286b720c6eb73c2b6286f93161: Status 404 returned error can't find the container with id 63cb668f6be31deb92c0bf5002b88716c90346286b720c6eb73c2b6286f93161
Oct 30 21:09:04.921047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:04.921024    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:09:05.037140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:05.037098    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td" event={"ID":"3837a8c5-b627-4a55-b2bc-7cfcede7214c","Type":"ContainerStarted","Data":"63cb668f6be31deb92c0bf5002b88716c90346286b720c6eb73c2b6286f93161"}
Oct 30 21:09:06.043425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:06.043392    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td" event={"ID":"3837a8c5-b627-4a55-b2bc-7cfcede7214c","Type":"ContainerStarted","Data":"18b081948aa0c450c33ffa7e168ef75fa9b7f49efc3fb9a73e0e1f283af8ef72"}
Oct 30 21:09:06.043879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:06.043536    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:06.083697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:06.083643    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td" podStartSLOduration=1.790725423 podStartE2EDuration="2.083626166s" totalImagesPullingTime="292.900743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:06.083070458 +0000 UTC m=+489.494609894" watchObservedRunningTime="2026-10-30 21:09:06.083626166 +0000 UTC m=+489.495165597"
Oct 30 21:09:17.050725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.050692    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-9b9d764bc-9m9td"
Oct 30 21:09:17.126284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.126245    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:09:17.126522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.126461    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" podUID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" containerName="manager" containerID="cri-o://642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552" gracePeriod=30
Oct 30 21:09:17.384713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.384687    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:09:17.498741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.498705    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\" (UniqueName: \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\") pod \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") "
Oct 30 21:09:17.498926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.498769    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\") pod \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\" (UID: \"c4a65e60-e7c3-4485-9bd6-171a7226be7a\") "
Oct 30 21:09:17.500887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.500864    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw" pod "c4a65e60-e7c3-4485-9bd6-171a7226be7a" (UID: "c4a65e60-e7c3-4485-9bd6-171a7226be7a"). InnerVolumeSpecName "kube-api-access-qqbdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:17.501062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.501039    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs" pod "c4a65e60-e7c3-4485-9bd6-171a7226be7a" (UID: "c4a65e60-e7c3-4485-9bd6-171a7226be7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:17.599736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.599701    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbdw\" (UniqueName: \"kubernetes.io/projected/c4a65e60-e7c3-4485-9bd6-171a7226be7a-kube-api-access-qqbdw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:09:17.599736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:17.599733    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a65e60-e7c3-4485-9bd6-171a7226be7a-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:09:18.103149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.103117    7037 generic.go:356] "Generic (PLEG): container finished" podID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" containerID="642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552" exitCode=0
Oct 30 21:09:18.103657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.103177    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj"
Oct 30 21:09:18.103657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.103210    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" event={"ID":"c4a65e60-e7c3-4485-9bd6-171a7226be7a","Type":"ContainerDied","Data":"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"}
Oct 30 21:09:18.103657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.103281    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-6gcwj" event={"ID":"c4a65e60-e7c3-4485-9bd6-171a7226be7a","Type":"ContainerDied","Data":"e1f17345d7844c849ed8e1b4178b39fae3c41514c980a6e980f2b3d494ba42c2"}
Oct 30 21:09:18.103657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.103316    7037 scope.go:122] "RemoveContainer" containerID="642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"
Oct 30 21:09:18.112904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.112886    7037 scope.go:122] "RemoveContainer" containerID="642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"
Oct 30 21:09:18.113344 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:09:18.113264    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552\": container with ID starting with 642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552 not found: ID does not exist" containerID="642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"
Oct 30 21:09:18.113344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.113308    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552"} err="failed to get container status \"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552\": rpc error: code = NotFound desc = could not find container \"642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552\": container with ID starting with 642451838bf387ee4f6d3a1936f49b8d5e3c46b4e58b5b708bdbead8c9769552 not found: ID does not exist"
Oct 30 21:09:18.152766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.152737    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:09:18.164951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.164929    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-6gcwj"]
Oct 30 21:09:18.706668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:18.706633    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" path="/var/lib/kubelet/pods/c4a65e60-e7c3-4485-9bd6-171a7226be7a/volumes"
Oct 30 21:09:33.670626 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:09:33.670579    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:09:44.656693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.656660    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:09:44.657357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.657337    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" containerName="manager"
Oct 30 21:09:44.657357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.657357    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" containerName="manager"
Oct 30 21:09:44.657460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.657440    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a65e60-e7c3-4485-9bd6-171a7226be7a" containerName="manager"
Oct 30 21:09:44.659726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.659703    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663342    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663375    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663419    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663480    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663583    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663608    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663631    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663726    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.663793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.663776    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dcr\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.689972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.689948    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-58fea9-openshift-default-dockercfg-2wlz5\""
Oct 30 21:09:44.691437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.691415    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:09:44.765078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765042    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765084    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765118    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765153    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765194    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765213    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765242    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765276    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765325    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dcr\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765534    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765589    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765675    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.765921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.765897    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.766024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.766004    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.767758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.767729    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.767884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.767865    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.775275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.775244    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:09:44.778217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.778185    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.792126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.792028    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.792352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.792333    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dcr\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.794047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.794027    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-43c070-openshift-default-dockercfg-srjfd\""
Oct 30 21:09:44.801085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.801065    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:09:44.866227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866189    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866302    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866341    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866367    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866479    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866521    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866611    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866657    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kz9\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.866775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.866711    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967466    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967551    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967620    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967646    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967674    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967713    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967746    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.967797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967777    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967803    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kz9\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.967958    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.968013    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.968266    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.968524    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.968899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.968632    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.970682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.970652    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.971570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.971549    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.972597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.972574    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:44.987741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.987712    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:44.987930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:44.987907    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kz9\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:45.121059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.121025    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:45.214395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.214362    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:09:45.217101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.217086    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.239209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.239181    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-79477e-openshift-default-dockercfg-448fh\""
Oct 30 21:09:45.240031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.240011    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:09:45.271534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271499    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271585    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4xr\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271651    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271703    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271721    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271737    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271756    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.271997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.271979    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.272058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.272038    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.288142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.288119    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:09:45.288476 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:45.288456    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3db193_1d90_49e9_9013_59bcbf381888.slice/crio-661fbd7a3d2d962083be88f63d60cc59b7a9da1fd9e359c7ab744f6e9cacb9e7 WatchSource:0}: Error finding container 661fbd7a3d2d962083be88f63d60cc59b7a9da1fd9e359c7ab744f6e9cacb9e7: Status 404 returned error can't find the container with id 661fbd7a3d2d962083be88f63d60cc59b7a9da1fd9e359c7ab744f6e9cacb9e7
Oct 30 21:09:45.290634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.290605    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.290708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.290667    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.290708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.290696    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.373630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373582    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.373630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373648    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.373928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373678    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.373928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373708    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.373928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373859    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.373928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373913    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373954    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.373996    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4xr\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.374031    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.374456    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.374751    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.374924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.374905    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.375119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.375098    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.375772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.375747    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.377375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.377347    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.380094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.380073    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.394194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.394167    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.394901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.394866    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4xr\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.417328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.417298    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:09:45.419104 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:45.419080    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10424c1b_c3f5_449b_8369_c6acb60ba8d2.slice/crio-74c8ad245303d19ac7bff314bf5470e15db5d70c12759f155da1f957351cd145 WatchSource:0}: Error finding container 74c8ad245303d19ac7bff314bf5470e15db5d70c12759f155da1f957351cd145: Status 404 returned error can't find the container with id 74c8ad245303d19ac7bff314bf5470e15db5d70c12759f155da1f957351cd145
Oct 30 21:09:45.422056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.422025    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.422126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.422092    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.422162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.422125    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.489703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.489675    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:09:45.493967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.493933    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.514723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.514694    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:09:45.515580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.515518    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ed088d-openshift-default-dockercfg-xqfc5\""
Oct 30 21:09:45.532554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.532523    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577479    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577607    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577665    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577698    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577740    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577778    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577823    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577851    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.578187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.577952    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wst22\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679149    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wst22\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679208    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679239    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679284    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679506    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679564    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679608    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.679665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679656    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679685    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679713    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.679907    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.680184    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.680227    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.680738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.680720    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.682333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.682299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.683024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.683002    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.698635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.698612    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.698944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.698926    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wst22\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.807984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.807905    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:45.843895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.843864    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:09:45.851054 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:45.851026    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef6aea4_d75b_4622_8020_81773b7a77d5.slice/crio-bdfef0319d4f6cd5732f186be1cc62ca32d2c2e72afa060a323814e2a27e6ce3 WatchSource:0}: Error finding container bdfef0319d4f6cd5732f186be1cc62ca32d2c2e72afa060a323814e2a27e6ce3: Status 404 returned error can't find the container with id bdfef0319d4f6cd5732f186be1cc62ca32d2c2e72afa060a323814e2a27e6ce3
Oct 30 21:09:45.855978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.855939    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.856090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.856013    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:45.856090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:45.856048    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:46.125231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.125205    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:09:46.125905 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:46.125881    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c470506_029a_404a_9441_7f7cd07dd3fc.slice/crio-853febaf1975584c9892d58bfa488f827a7aa45ee74fc2557dadd731791ebeee WatchSource:0}: Error finding container 853febaf1975584c9892d58bfa488f827a7aa45ee74fc2557dadd731791ebeee: Status 404 returned error can't find the container with id 853febaf1975584c9892d58bfa488f827a7aa45ee74fc2557dadd731791ebeee
Oct 30 21:09:46.128921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.128881    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:46.129054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.128979    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:46.129054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.129037    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:09:46.239030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.238997    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerStarted","Data":"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"}
Oct 30 21:09:46.239195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.239058    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerStarted","Data":"74c8ad245303d19ac7bff314bf5470e15db5d70c12759f155da1f957351cd145"}
Oct 30 21:09:46.240664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.240634    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerStarted","Data":"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"}
Oct 30 21:09:46.240799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.240682    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerStarted","Data":"bdfef0319d4f6cd5732f186be1cc62ca32d2c2e72afa060a323814e2a27e6ce3"}
Oct 30 21:09:46.242222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.242198    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerStarted","Data":"21e14e7ec1d004244d3e73633e4e4c3eb8f7d849dee84fbe7773f79718c3bb42"}
Oct 30 21:09:46.242327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.242250    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerStarted","Data":"853febaf1975584c9892d58bfa488f827a7aa45ee74fc2557dadd731791ebeee"}
Oct 30 21:09:46.243985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.243962    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerStarted","Data":"25406431f06b088d1ac0c485331a91be4aea9a82457b145c3c81fb3ddda76410"}
Oct 30 21:09:46.244078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.244006    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerStarted","Data":"661fbd7a3d2d962083be88f63d60cc59b7a9da1fd9e359c7ab744f6e9cacb9e7"}
Oct 30 21:09:46.281625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.281564    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podStartSLOduration=2.281549476 podStartE2EDuration="2.281549476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.279795927 +0000 UTC m=+529.691335364" watchObservedRunningTime="2026-10-30 21:09:46.281549476 +0000 UTC m=+529.693088910"
Oct 30 21:09:46.322778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.322727    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podStartSLOduration=2.322713306 podStartE2EDuration="2.322713306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.321830319 +0000 UTC m=+529.733369756" watchObservedRunningTime="2026-10-30 21:09:46.322713306 +0000 UTC m=+529.734252743"
Oct 30 21:09:46.361588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.361533    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podStartSLOduration=1.361520744 podStartE2EDuration="1.361520744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.360788112 +0000 UTC m=+529.772327549" watchObservedRunningTime="2026-10-30 21:09:46.361520744 +0000 UTC m=+529.773060180"
Oct 30 21:09:46.402844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.402751    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podStartSLOduration=1.402737479 podStartE2EDuration="1.402737479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.401997612 +0000 UTC m=+529.813537049" watchObservedRunningTime="2026-10-30 21:09:46.402737479 +0000 UTC m=+529.814276916"
Oct 30 21:09:46.535096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.533685    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:46.543109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.543076    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:46.808173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.808103    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:46.973691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.973653    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:46.979327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:46.979300    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:47.121547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.121509    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:47.126994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.126966    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:47.250201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250169    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:47.250201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250224    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:47.250461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250252    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:47.250461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250442    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:09:47.250595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250519    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:09:47.250701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.250681    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:09:47.813371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:47.813342    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:48.255003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:48.254971    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:48.255865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:48.255845    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:09:50.784810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.784765    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:09:50.788940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.788923    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.805187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.805155    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-56ac74\""
Oct 30 21:09:50.807682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.807655    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:09:50.817920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.817895    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-56ac74-dockercfg-dbhqb\""
Oct 30 21:09:50.839695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.839657    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdfh\" (UniqueName: \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.839846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.839730    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.941019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.940975    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdfh\" (UniqueName: \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.941211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.941059    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.941799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.941777    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:50.960433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:50.960400    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdfh\" (UniqueName: \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\") pod \"payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:51.102755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:51.102721    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:51.400521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:51.400458    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:09:51.401023 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:51.400996    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7716237_5b0f_43aa_824f_1a74121d9a29.slice/crio-a1cc138579f247ec2b02a29dd9ff506fac9e6921b042e6f2107bbb82956fea1a WatchSource:0}: Error finding container a1cc138579f247ec2b02a29dd9ff506fac9e6921b042e6f2107bbb82956fea1a: Status 404 returned error can't find the container with id a1cc138579f247ec2b02a29dd9ff506fac9e6921b042e6f2107bbb82956fea1a
Oct 30 21:09:52.278369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.278333    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" event={"ID":"e7716237-5b0f-43aa-824f-1a74121d9a29","Type":"ContainerStarted","Data":"a1cc138579f247ec2b02a29dd9ff506fac9e6921b042e6f2107bbb82956fea1a"}
Oct 30 21:09:52.905375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.905336    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:09:52.914858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.914834    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:52.929903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.929872    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:09:52.938506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.938473    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-58fea9\""
Oct 30 21:09:52.938632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.938566    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-58fea9-dockercfg-qhpf2\""
Oct 30 21:09:52.962391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.962359    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:52.962670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:52.962628    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfhb\" (UniqueName: \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.064360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.063816    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfhb\" (UniqueName: \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.064719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.064449    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.065372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.065349    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.088393 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.088361    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfhb\" (UniqueName: \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\") pod \"payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.236095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.236005    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:53.538828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:53.538803    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:09:53.539459 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:53.539432    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d9b503_6346_439a_81be_4fb943579121.slice/crio-d10947e1888c650d97e7d823fbb9b530bff3ed9413fbba01e7f6d100350b7125 WatchSource:0}: Error finding container d10947e1888c650d97e7d823fbb9b530bff3ed9413fbba01e7f6d100350b7125: Status 404 returned error can't find the container with id d10947e1888c650d97e7d823fbb9b530bff3ed9413fbba01e7f6d100350b7125
Oct 30 21:09:54.291103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.291064    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" event={"ID":"e7716237-5b0f-43aa-824f-1a74121d9a29","Type":"ContainerStarted","Data":"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"}
Oct 30 21:09:54.291299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.291137    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:09:54.292648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.292621    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" event={"ID":"25d9b503-6346-439a-81be-4fb943579121","Type":"ContainerStarted","Data":"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"}
Oct 30 21:09:54.292755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.292671    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" event={"ID":"25d9b503-6346-439a-81be-4fb943579121","Type":"ContainerStarted","Data":"d10947e1888c650d97e7d823fbb9b530bff3ed9413fbba01e7f6d100350b7125"}
Oct 30 21:09:54.292794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.292750    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:09:54.333050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.332985    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" podStartSLOduration=2.324336115 podStartE2EDuration="4.332966473s" totalImagesPullingTime="2.008630358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:54.331016299 +0000 UTC m=+537.742555735" watchObservedRunningTime="2026-10-30 21:09:54.332966473 +0000 UTC m=+537.744505913"
Oct 30 21:09:54.373189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:54.373136    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" podStartSLOduration=2.373121569 podStartE2EDuration="2.373121569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:54.370647783 +0000 UTC m=+537.782187219" watchObservedRunningTime="2026-10-30 21:09:54.373121569 +0000 UTC m=+537.784661005"
Oct 30 21:09:55.017908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.017868    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:09:55.022202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.022171    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.049297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.049265    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-43c070\""
Oct 30 21:09:55.049297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.049276    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-43c070-dockercfg-fhzk4\""
Oct 30 21:09:55.051230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.051211    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:09:55.075856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.075825    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:09:55.080278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.080257    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.086064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.086026    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzq7\" (UniqueName: \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.086198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.086088    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.097830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.097803    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:09:55.187417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.187381    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v8k\" (UniqueName: \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.187635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.187598    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.187713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.187655    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzq7\" (UniqueName: \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.187773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.187713    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.188415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.188396    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.212236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.212204    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzq7\" (UniqueName: \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\") pod \"payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.289341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.289255    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v8k\" (UniqueName: \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.289341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.289342    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.290047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.290028    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.311568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.311539    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v8k\" (UniqueName: \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\") pod \"payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.336197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.336167    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:55.393798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.393772    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:55.633175 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.633137    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:09:55.633349 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:55.633321    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a4337b_76c0_4bbb_af4b_4098ac252864.slice/crio-46f15bab3d6418dd58fc8910d668ec863684d8d39929672f774fb9b097b54496 WatchSource:0}: Error finding container 46f15bab3d6418dd58fc8910d668ec863684d8d39929672f774fb9b097b54496: Status 404 returned error can't find the container with id 46f15bab3d6418dd58fc8910d668ec863684d8d39929672f774fb9b097b54496
Oct 30 21:09:55.697379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.697358    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:09:55.698667 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:55.698638    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a1927f_7fa5_4080_9386_e0227138f6bc.slice/crio-5ecfaea8895cb029cfd5c5fd65147f3aade5d7b7bd77c3196731e683d87a1a07 WatchSource:0}: Error finding container 5ecfaea8895cb029cfd5c5fd65147f3aade5d7b7bd77c3196731e683d87a1a07: Status 404 returned error can't find the container with id 5ecfaea8895cb029cfd5c5fd65147f3aade5d7b7bd77c3196731e683d87a1a07
Oct 30 21:09:55.788399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.788366    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:09:55.792551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.792520    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:55.817805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.817772    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ed088d\""
Oct 30 21:09:55.820065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.820038    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:09:55.857735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.857697    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:09:55.861647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.861620    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:55.866714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.866691    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ed088d-dockercfg-j7bn2\""
Oct 30 21:09:55.884801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.884735    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:09:55.895934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.895907    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:55.896074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.895947    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkv45\" (UniqueName: \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:55.896074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.896021    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzt9\" (UniqueName: \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:55.896074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.896066    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:55.997379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.997333    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:55.997622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.997401    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkv45\" (UniqueName: \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:55.997622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.997443    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzt9\" (UniqueName: \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:55.997622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:55.997510    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:56.001551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.001509    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:56.002607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.002577    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:56.023127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.023091    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkv45\" (UniqueName: \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\") pod \"payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:56.023834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.023813    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzt9\" (UniqueName: \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\") pod \"payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:56.108026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.107994    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:56.175724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.175698    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:56.305444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.305405    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" event={"ID":"f5a1927f-7fa5-4080-9386-e0227138f6bc","Type":"ContainerStarted","Data":"f538b97c8f33c33752478f9055243327ccc1829b2536cf409105fd1c73f2300b"}
Oct 30 21:09:56.305681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.305470    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" event={"ID":"f5a1927f-7fa5-4080-9386-e0227138f6bc","Type":"ContainerStarted","Data":"5ecfaea8895cb029cfd5c5fd65147f3aade5d7b7bd77c3196731e683d87a1a07"}
Oct 30 21:09:56.305681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.305514    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:09:56.307156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.307130    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" event={"ID":"11a4337b-76c0-4bbb-af4b-4098ac252864","Type":"ContainerStarted","Data":"59609f3f5bec595acbf7a7b1550ae4febcba6725f51d77e31dbab15861b83381"}
Oct 30 21:09:56.307272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.307191    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" event={"ID":"11a4337b-76c0-4bbb-af4b-4098ac252864","Type":"ContainerStarted","Data":"46f15bab3d6418dd58fc8910d668ec863684d8d39929672f774fb9b097b54496"}
Oct 30 21:09:56.307272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.307215    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:09:56.344256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.344208    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" podStartSLOduration=1.344196332 podStartE2EDuration="1.344196332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:56.343815933 +0000 UTC m=+539.755355369" watchObservedRunningTime="2026-10-30 21:09:56.344196332 +0000 UTC m=+539.755735770"
Oct 30 21:09:56.384319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.384264    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" podStartSLOduration=2.384252135 podStartE2EDuration="2.384252135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:56.382469112 +0000 UTC m=+539.794008548" watchObservedRunningTime="2026-10-30 21:09:56.384252135 +0000 UTC m=+539.795791571"
Oct 30 21:09:56.399831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.399802    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:09:56.400448 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:56.400420    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85b6942_19ff_4eb5_8ce3_46795a93ef34.slice/crio-9593b9ecdc0d2bfe56b7bc38968ffbb8ae9c673b389743557f8380d4ed3dca40 WatchSource:0}: Error finding container 9593b9ecdc0d2bfe56b7bc38968ffbb8ae9c673b389743557f8380d4ed3dca40: Status 404 returned error can't find the container with id 9593b9ecdc0d2bfe56b7bc38968ffbb8ae9c673b389743557f8380d4ed3dca40
Oct 30 21:09:56.472294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:56.472267    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:09:56.472721 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:56.472697    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dbfe4e_34b9_48a0_a9e7_f4b349d1271a.slice/crio-f733bdff1e7d98a701106e4155d1a3527c0bdccdf462e2da77de9023be267e6e WatchSource:0}: Error finding container f733bdff1e7d98a701106e4155d1a3527c0bdccdf462e2da77de9023be267e6e: Status 404 returned error can't find the container with id f733bdff1e7d98a701106e4155d1a3527c0bdccdf462e2da77de9023be267e6e
Oct 30 21:09:57.317808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.317762    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" event={"ID":"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a","Type":"ContainerStarted","Data":"6e30ad32ced8070eb4b06d9740b6b7b0d49e7903fc605a49e931c582fa349555"}
Oct 30 21:09:57.318328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.317868    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" event={"ID":"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a","Type":"ContainerStarted","Data":"f733bdff1e7d98a701106e4155d1a3527c0bdccdf462e2da77de9023be267e6e"}
Oct 30 21:09:57.318328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.317991    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:09:57.319642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.319612    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" event={"ID":"c85b6942-19ff-4eb5-8ce3-46795a93ef34","Type":"ContainerStarted","Data":"4fd0b515b04da79e8640e512b811189578b3bdba43550a1fd667e6b45372d508"}
Oct 30 21:09:57.319765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.319663    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" event={"ID":"c85b6942-19ff-4eb5-8ce3-46795a93ef34","Type":"ContainerStarted","Data":"9593b9ecdc0d2bfe56b7bc38968ffbb8ae9c673b389743557f8380d4ed3dca40"}
Oct 30 21:09:57.358392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.358328    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" podStartSLOduration=2.358310096 podStartE2EDuration="2.358310096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:57.357714885 +0000 UTC m=+540.769254322" watchObservedRunningTime="2026-10-30 21:09:57.358310096 +0000 UTC m=+540.769849536"
Oct 30 21:09:57.377864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.377832    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:09:57.383124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.383101    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.398303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.398258    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" podStartSLOduration=2.398244579 podStartE2EDuration="2.398244579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:57.397927483 +0000 UTC m=+540.809466921" watchObservedRunningTime="2026-10-30 21:09:57.398244579 +0000 UTC m=+540.809784028"
Oct 30 21:09:57.404326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.404300    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:09:57.404576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.404559    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2cda44-dockercfg-27cbx\""
Oct 30 21:09:57.418682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.418641    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2cda44\""
Oct 30 21:09:57.526403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.526354    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm88\" (UniqueName: \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.526623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.526482    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.627396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.627304    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm88\" (UniqueName: \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.627690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.627665    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.628933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.628908    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.649516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.649445    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm88\" (UniqueName: \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\") pod \"payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.697689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.697653    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:57.993655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:57.993625    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:09:57.994391 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:09:57.994363    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a226687_037d_47f7_a6e4_2899c18a7ee9.slice/crio-8725b6b0c4d5b6065778b629cead0af569c8c66f199c7b9c2fc533dff2b1c825 WatchSource:0}: Error finding container 8725b6b0c4d5b6065778b629cead0af569c8c66f199c7b9c2fc533dff2b1c825: Status 404 returned error can't find the container with id 8725b6b0c4d5b6065778b629cead0af569c8c66f199c7b9c2fc533dff2b1c825
Oct 30 21:09:58.334372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:58.334334    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" event={"ID":"6a226687-037d-47f7-a6e4-2899c18a7ee9","Type":"ContainerStarted","Data":"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"}
Oct 30 21:09:58.335234 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:58.335199    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:09:58.335366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:58.335247    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" event={"ID":"6a226687-037d-47f7-a6e4-2899c18a7ee9","Type":"ContainerStarted","Data":"8725b6b0c4d5b6065778b629cead0af569c8c66f199c7b9c2fc533dff2b1c825"}
Oct 30 21:09:58.335366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:58.335279    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:09:58.376289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:58.376234    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" podStartSLOduration=1.376218697 podStartE2EDuration="1.376218697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:58.374906983 +0000 UTC m=+541.786446419" watchObservedRunningTime="2026-10-30 21:09:58.376218697 +0000 UTC m=+541.787758134"
Oct 30 21:09:59.244811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.244771    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:09:59.248832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.248813    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.279853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.279816    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-43c070-serving-cert\""
Oct 30 21:09:59.280104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.280075    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:09:59.280104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.280106    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:09:59.280295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.280160    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:09:59.280295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.280246    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pdl4\""
Oct 30 21:09:59.282155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.282134    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:09:59.346700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.346665    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdw9\" (UniqueName: \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.347144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.346715    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.347144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.346830    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.347144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.346927    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.448799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.448758    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.449047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.448927    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdw9\" (UniqueName: \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.449047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.449031    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.449153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.449137    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.449413 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:09:59.449382    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-serving-cert: secret "maas-api-e2e-worker-w5-43c070-serving-cert" not found
Oct 30 21:09:59.450545 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:09:59.449482    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls podName:8e36be97-86a1-49d3-b9b9-290c7395730d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:59.949454574 +0000 UTC m=+543.360994005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls") pod "maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" (UID: "8e36be97-86a1-49d3-b9b9-290c7395730d") : secret "maas-api-e2e-worker-w5-43c070-serving-cert" not found
Oct 30 21:09:59.450545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.450346    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.452562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.452541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.472804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.472773    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdw9\" (UniqueName: \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.953226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.953183    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:09:59.955709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:09:59.955683    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:10:00.162041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:00.162001    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:10:00.458640 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:10:00.458606    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e36be97_86a1_49d3_b9b9_290c7395730d.slice/crio-9f54d7814990df65b17dd9c887b691d67411ceb3073f24b0a85ae78e9547b096 WatchSource:0}: Error finding container 9f54d7814990df65b17dd9c887b691d67411ceb3073f24b0a85ae78e9547b096: Status 404 returned error can't find the container with id 9f54d7814990df65b17dd9c887b691d67411ceb3073f24b0a85ae78e9547b096
Oct 30 21:10:00.459285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:00.459259    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:10:01.358587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:01.358548    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" event={"ID":"8e36be97-86a1-49d3-b9b9-290c7395730d","Type":"ContainerStarted","Data":"9f54d7814990df65b17dd9c887b691d67411ceb3073f24b0a85ae78e9547b096"}
Oct 30 21:10:02.367067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:02.367028    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" event={"ID":"8e36be97-86a1-49d3-b9b9-290c7395730d","Type":"ContainerStarted","Data":"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"}
Oct 30 21:10:02.367506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:02.367147    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:10:02.406651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:02.406600    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" podStartSLOduration=1.886146111 podStartE2EDuration="3.406586213s" totalImagesPullingTime="1.520440102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.405745908 +0000 UTC m=+545.817285346" watchObservedRunningTime="2026-10-30 21:10:02.406586213 +0000 UTC m=+545.818125649"
Oct 30 21:10:03.789122 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:10:03.789080    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:10:05.303848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:05.303818    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:10:05.308091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:05.308035    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:10:07.322857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:07.322678    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:10:07.323219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:07.323026    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:10:08.352753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:08.352715    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:10:08.404917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:08.404886    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:10:09.342341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:09.342311    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:10:09.342927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:09.342903    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:10:15.603137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.603073    7037 generic.go:356] "Generic (PLEG): container finished" podID="bf3db193-1d90-49e9-9013-59bcbf381888" containerID="25406431f06b088d1ac0c485331a91be4aea9a82457b145c3c81fb3ddda76410" exitCode=0
Oct 30 21:10:15.603585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.603276    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerDied","Data":"25406431f06b088d1ac0c485331a91be4aea9a82457b145c3c81fb3ddda76410"}
Oct 30 21:10:15.604181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.603786    7037 scope.go:122] "RemoveContainer" containerID="25406431f06b088d1ac0c485331a91be4aea9a82457b145c3c81fb3ddda76410"
Oct 30 21:10:15.608930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.606212    7037 generic.go:356] "Generic (PLEG): container finished" podID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerID="a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0" exitCode=0
Oct 30 21:10:15.608930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.606301    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerDied","Data":"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"}
Oct 30 21:10:15.608930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.606702    7037 scope.go:122] "RemoveContainer" containerID="a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"
Oct 30 21:10:15.609204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.609168    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:15.609270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.609254    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:15.609323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.609299    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:15.610376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.610345    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:15.610524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.610407    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:15.610524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:15.610443    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:16.614180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.614138    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerStarted","Data":"22779482cbffc757ff315e6bbe4543dc7a8e17fee21e396b16b33a37ccab5a30"}
Oct 30 21:10:16.616624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.616598    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerStarted","Data":"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"}
Oct 30 21:10:16.618920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.618893    7037 generic.go:356] "Generic (PLEG): container finished" podID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerID="a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e" exitCode=0
Oct 30 21:10:16.618983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.618924    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerDied","Data":"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"}
Oct 30 21:10:16.619189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.619175    7037 scope.go:122] "RemoveContainer" containerID="a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"
Oct 30 21:10:16.620309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.620281    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:16.620358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.620345    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:16.620393 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.620375    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:16.977623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.977083    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:10:16.977623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.977224    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:16.977623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:16.977268    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:17.122152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.122124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:10:17.126524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.123485    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:10:17.126524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.123556    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:10:17.253580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.252613    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:10:17.253580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.252737    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:10:17.253580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.252823    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:10:17.658521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.657649    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerStarted","Data":"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"}
Oct 30 21:10:17.981565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.981436    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:17.982612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:17.982568    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:18.122008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.121975    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:10:18.122236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.122213    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:10:18.544514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.538778    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:10:18.544514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.538913    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:10:18.544514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.538974    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 21:10:18.973512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.972991    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:18.973512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:18.973057    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:19.121873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.121836    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:10:19.129570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.121899    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:10:19.305828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.305722    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:10:19.305828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.305789    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:10:19.541973 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.541935    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:10:19.542295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.542265    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 21:10:19.974516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.973567    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:19.974516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:19.973630    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:20.122872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.121959    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:10:20.122872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.122024    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:10:20.535404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.534913    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:10:20.535404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.534995    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 21:10:20.976913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.976875    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:20.984519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:20.977461    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:21.122846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.122568    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:10:21.122846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.122627    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:10:21.592513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.585606    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:10:21.592513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.585673    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 21:10:21.974797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.974715    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:10:21.974797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:21.974777    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 21:10:22.253663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.252444    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:10:22.269521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.269385    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:10:22.535968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.535879    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:10:22.536244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.536214    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 30 21:10:22.979476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.979444    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:10:22.980786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:22.980762    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:10:23.538015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:23.537986    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:10:23.538503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:23.538467    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:10:23.539141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:23.539124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:10:24.730821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.730787    7037 generic.go:356] "Generic (PLEG): container finished" podID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerID="21e14e7ec1d004244d3e73633e4e4c3eb8f7d849dee84fbe7773f79718c3bb42" exitCode=0
Oct 30 21:10:24.731251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.730866    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerDied","Data":"21e14e7ec1d004244d3e73633e4e4c3eb8f7d849dee84fbe7773f79718c3bb42"}
Oct 30 21:10:24.731325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.731310    7037 scope.go:122] "RemoveContainer" containerID="21e14e7ec1d004244d3e73633e4e4c3eb8f7d849dee84fbe7773f79718c3bb42"
Oct 30 21:10:24.732524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.732479    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:24.732593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.732563    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:24.732631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:24.732596    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:10:25.758458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:25.758423    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerStarted","Data":"376e9171ee736e09854c323716fdc6c30aed837c5cb5b2c5f43a8f55a670357e"}
Oct 30 21:10:25.808664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:25.808630    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:10:25.808863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:25.808789    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:25.808863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:25.808829    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:26.808299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:26.808262    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:26.808829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:26.808322    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:27.808442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:27.808402    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:27.808988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:27.808479    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:28.811259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:28.811227    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:28.811675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:28.811291    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:29.808233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:29.808174    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:29.808233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:29.808232    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:30.808291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:30.808250    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:10:30.808841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:30.808308    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 21:10:31.813363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:31.813328    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:10:31.813829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:31.813810    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:10:31.814483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:31.814462    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:10:33.673999 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:10:33.673955    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:10:38.673901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.673857    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:10:38.677787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.677761    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.695956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.695926    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:10:38.706394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.706364    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"simulator-w0-58fea9-kserve-self-signed-certs\""
Oct 30 21:10:38.706571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.706406    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"default-dockercfg-wmzx2\""
Oct 30 21:10:38.708979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.708954    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"openshift-service-ca.crt\""
Oct 30 21:10:38.709098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.709063    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"kube-root-ca.crt\""
Oct 30 21:10:38.818733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.818690    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.818890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.818823    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.819035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.819001    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hvz\" (UniqueName: \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.819109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.819092    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.819152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.819141    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.819246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.819230    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920185    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920259    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920335    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hvz\" (UniqueName: \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920370    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920415    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920453    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920735    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.920817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.920762    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.921279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.921251    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.923297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.923275    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.923433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.923409    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.942584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.942520    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hvz\" (UniqueName: \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\") pod \"simulator-w0-58fea9-kserve-7f5d684569-w4kpc\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") " pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:38.992380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:38.992345    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:39.290331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:39.290295    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:10:39.290459 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:10:39.290372    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad803c3d_e2e8_400f_995b_c81cd7d60772.slice/crio-70204344e7cb23a572dec23fc8c97a44afd2e7c6e17ed4646f1637a6248cfd22 WatchSource:0}: Error finding container 70204344e7cb23a572dec23fc8c97a44afd2e7c6e17ed4646f1637a6248cfd22: Status 404 returned error can't find the container with id 70204344e7cb23a572dec23fc8c97a44afd2e7c6e17ed4646f1637a6248cfd22
Oct 30 21:10:39.847174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:39.847134    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" event={"ID":"ad803c3d-e2e8-400f-995b-c81cd7d60772","Type":"ContainerStarted","Data":"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"}
Oct 30 21:10:39.847890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:39.847842    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" event={"ID":"ad803c3d-e2e8-400f-995b-c81cd7d60772","Type":"ContainerStarted","Data":"70204344e7cb23a572dec23fc8c97a44afd2e7c6e17ed4646f1637a6248cfd22"}
Oct 30 21:10:39.889552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:39.889470    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podStartSLOduration=1.889454078 podStartE2EDuration="1.889454078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:39.888363421 +0000 UTC m=+583.299902858" watchObservedRunningTime="2026-10-30 21:10:39.889454078 +0000 UTC m=+583.300993514"
Oct 30 21:10:39.993085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:39.993030    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:46.945746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.945703    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:10:46.950698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.950669    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:46.969562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.969518    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:10:46.979104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.979070    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"default-dockercfg-qz7t2\""
Oct 30 21:10:46.979104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.979076    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"simulator-w2-2cda44-kserve-self-signed-certs\""
Oct 30 21:10:46.979916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.979899    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"openshift-service-ca.crt\""
Oct 30 21:10:46.980134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:46.980117    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"kube-root-ca.crt\""
Oct 30 21:10:47.004070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.004030    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.004376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.004354    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.004514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.004480    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.004649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.004635    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.004765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.004752    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.005076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.005039    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xn8\" (UniqueName: \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106245    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106307    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106326    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106348    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106371    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.106543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106413    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xn8\" (UniqueName: \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.107245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106765    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.107245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106813    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.107245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.106940    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.108674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.108637    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.109267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.109249    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.127355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.127317    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xn8\" (UniqueName: \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\") pod \"simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") " pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.267962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.267874    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:47.576610 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:10:47.576576    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c385729_0387_4c4e_af25_12f078af3177.slice/crio-eb00985bd3c5ef08ba5cb24e109adfbeb5ac13d5da939db8bab181cca04afc89 WatchSource:0}: Error finding container eb00985bd3c5ef08ba5cb24e109adfbeb5ac13d5da939db8bab181cca04afc89: Status 404 returned error can't find the container with id eb00985bd3c5ef08ba5cb24e109adfbeb5ac13d5da939db8bab181cca04afc89
Oct 30 21:10:47.576908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.576885    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:10:47.892873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.892778    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" event={"ID":"6c385729-0387-4c4e-af25-12f078af3177","Type":"ContainerStarted","Data":"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"}
Oct 30 21:10:47.892873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.892826    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" event={"ID":"6c385729-0387-4c4e-af25-12f078af3177","Type":"ContainerStarted","Data":"eb00985bd3c5ef08ba5cb24e109adfbeb5ac13d5da939db8bab181cca04afc89"}
Oct 30 21:10:47.941863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:47.941810    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podStartSLOduration=1.9417954370000001 podStartE2EDuration="1.941795437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:47.940185426 +0000 UTC m=+591.351724859" watchObservedRunningTime="2026-10-30 21:10:47.941795437 +0000 UTC m=+591.353334874"
Oct 30 21:10:48.268298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:48.268213    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:49.002104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:49.002067    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:49.014768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:49.014743    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:49.915653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:49.915621    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:10:57.268974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:57.268938    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:57.282247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:57.282204    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:10:57.962121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:10:57.962087    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:11:01.252286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.252248    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:11:01.257563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.257530    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.273334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.273305    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:11:01.283605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.283576    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"premium-w4-839163-kserve-self-signed-certs\""
Oct 30 21:11:01.283605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.283596    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"default-dockercfg-xgml5\""
Oct 30 21:11:01.286415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.286389    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"openshift-service-ca.crt\""
Oct 30 21:11:01.286553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.286432    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"kube-root-ca.crt\""
Oct 30 21:11:01.408007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.407967    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.408007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.408007    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.408214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.408040    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.408214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.408134    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.408285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.408227    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.408285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.408268    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqkl\" (UniqueName: \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.509965    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510001    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510030    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510053    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510093    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510139    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqkl\" (UniqueName: \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510436    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510503    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.510702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.510572    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.512350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.512329    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.512748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.512725    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.529958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.529927    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqkl\" (UniqueName: \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\") pod \"premium-w4-839163-kserve-8784b6c46-bf7pt\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") " pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.574711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.574679    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:01.871322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.871285    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:11:01.871919 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:11:01.871895    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda9c767d_9290_4cde_87d3_0d3bb6ecef8c.slice/crio-0a3b40c335a735910f3114f17e2bbe4d5d4341f24462692f8a67ea1511b8f8f0 WatchSource:0}: Error finding container 0a3b40c335a735910f3114f17e2bbe4d5d4341f24462692f8a67ea1511b8f8f0: Status 404 returned error can't find the container with id 0a3b40c335a735910f3114f17e2bbe4d5d4341f24462692f8a67ea1511b8f8f0
Oct 30 21:11:01.976577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.976539    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" event={"ID":"da9c767d-9290-4cde-87d3-0d3bb6ecef8c","Type":"ContainerStarted","Data":"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"}
Oct 30 21:11:01.976736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:01.976591    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" event={"ID":"da9c767d-9290-4cde-87d3-0d3bb6ecef8c","Type":"ContainerStarted","Data":"0a3b40c335a735910f3114f17e2bbe4d5d4341f24462692f8a67ea1511b8f8f0"}
Oct 30 21:11:02.015955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:02.015899    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podStartSLOduration=1.015884594 podStartE2EDuration="1.015884594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:02.014296903 +0000 UTC m=+605.425836340" watchObservedRunningTime="2026-10-30 21:11:02.015884594 +0000 UTC m=+605.427424030"
Oct 30 21:11:02.574928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:02.574893    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:03.671903 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:11:03.671870    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:11:11.575867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:11.575835    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:11.588263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:11.588241    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:12.041175 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:12.041096    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:11:15.758696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.758661    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:11:15.763346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.763326    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.786229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.786204    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"openshift-service-ca.crt\""
Oct 30 21:11:15.786229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.786206    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"kube-root-ca.crt\""
Oct 30 21:11:15.799013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.798973    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"premium-w5-43c070-kserve-self-signed-certs\""
Oct 30 21:11:15.799179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.799047    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"default-dockercfg-w4c7h\""
Oct 30 21:11:15.803190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.803166    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:11:15.863030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.862997    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.863198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.863155    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.863269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.863232    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.863269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.863251    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.863372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.863298    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.863438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.863423    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bd8\" (UniqueName: \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964159    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964211    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964244    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964322    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bd8\" (UniqueName: \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964366    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.964483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964481    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.965029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.964996    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.965618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.965583    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.965869 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.965845    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.978655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.978598    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.994933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.991453    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:15.998547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:15.995121    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bd8\" (UniqueName: \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\") pod \"premium-w5-43c070-kserve-584cdcc7f8-dhcrt\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") " pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:16.079093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:16.079059    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:16.385855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:16.385751    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:11:16.393279 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:11:16.393251    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7700176f_5342_4db9_bc74_f86befc42f39.slice/crio-5a455d1e1380a201927ff8e66266252aee43b4775bfe78e6c67c31d79930d4f5 WatchSource:0}: Error finding container 5a455d1e1380a201927ff8e66266252aee43b4775bfe78e6c67c31d79930d4f5: Status 404 returned error can't find the container with id 5a455d1e1380a201927ff8e66266252aee43b4775bfe78e6c67c31d79930d4f5
Oct 30 21:11:17.060665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:17.060624    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" event={"ID":"7700176f-5342-4db9-bc74-f86befc42f39","Type":"ContainerStarted","Data":"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"}
Oct 30 21:11:17.061130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:17.060679    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" event={"ID":"7700176f-5342-4db9-bc74-f86befc42f39","Type":"ContainerStarted","Data":"5a455d1e1380a201927ff8e66266252aee43b4775bfe78e6c67c31d79930d4f5"}
Oct 30 21:11:17.079450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:17.079427    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:17.103325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:17.103273    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podStartSLOduration=2.103260335 podStartE2EDuration="2.103260335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:17.102026331 +0000 UTC m=+620.513565767" watchObservedRunningTime="2026-10-30 21:11:17.103260335 +0000 UTC m=+620.514799808"
Oct 30 21:11:26.081511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:26.080852    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:26.104364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:26.104330    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:26.140514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:26.139767    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:11:30.227261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.227216    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:11:30.233502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.233470    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.248711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.248682    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:11:30.258761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.258737    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"premium-w2-2cda44-kserve-self-signed-certs\""
Oct 30 21:11:30.345999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.345963    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.345999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.345999    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.346248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.346086    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.346248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.346112    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.346358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.346331    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.346414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.346370    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82s8\" (UniqueName: \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.447855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.447794    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82s8\" (UniqueName: \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.448274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.448239    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.448411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.448282    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.448411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.448333    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.448411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.448360    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.448928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.448902    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.449141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.449119    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.449424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.449404    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.449590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.449565    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.451300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.451273    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.451812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.451787    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.467265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.467238    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82s8\" (UniqueName: \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\") pod \"premium-w2-2cda44-kserve-5d4669cd8d-fb2n9\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") " pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.548930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.548846    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:30.844216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:30.844183    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:11:30.844690 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:11:30.844662    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode27f9f09_d48e_4b8b_b857_6d949cf7633c.slice/crio-2ffdb8c4b2c47fd2ab67b5e36bbc417bbefc09b8129962f0df91bfec76eff608 WatchSource:0}: Error finding container 2ffdb8c4b2c47fd2ab67b5e36bbc417bbefc09b8129962f0df91bfec76eff608: Status 404 returned error can't find the container with id 2ffdb8c4b2c47fd2ab67b5e36bbc417bbefc09b8129962f0df91bfec76eff608
Oct 30 21:11:31.148338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:31.148255    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" event={"ID":"e27f9f09-d48e-4b8b-b857-6d949cf7633c","Type":"ContainerStarted","Data":"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"}
Oct 30 21:11:31.148338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:31.148300    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" event={"ID":"e27f9f09-d48e-4b8b-b857-6d949cf7633c","Type":"ContainerStarted","Data":"2ffdb8c4b2c47fd2ab67b5e36bbc417bbefc09b8129962f0df91bfec76eff608"}
Oct 30 21:11:31.190783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:31.190729    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podStartSLOduration=1.190715402 podStartE2EDuration="1.190715402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:31.188372483 +0000 UTC m=+634.599911932" watchObservedRunningTime="2026-10-30 21:11:31.190715402 +0000 UTC m=+634.602254836"
Oct 30 21:11:31.549004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:31.548919    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:33.672369 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:11:33.672332    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:11:40.549331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:40.549245    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:40.562131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:40.562103    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:41.261336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:41.261299    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:11:55.292008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.291970    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"]
Oct 30 21:11:55.296377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.296355    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.314691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.314654    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"]
Oct 30 21:11:55.323816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.323764    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:55.323986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.323824    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:11:55.398457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.398417    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.398457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.398456    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5zn\" (UniqueName: \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.398813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.398784    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.499813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.499772    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.500006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.499832    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.500006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.499958    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5zn\" (UniqueName: \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.500677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.500650    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.502343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.502317    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.520741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.520705    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5zn\" (UniqueName: \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\") pod \"maas-api-revoke-sub-37a25340-rb242\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.617068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.617028    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:11:55.923563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:55.923526    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"]
Oct 30 21:11:55.924457 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:11:55.924427    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce075a04_45a2_4af0_ba64_2dddd9f9d962.slice/crio-6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5 WatchSource:0}: Error finding container 6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5: Status 404 returned error can't find the container with id 6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5
Oct 30 21:11:56.351541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:56.351483    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242" event={"ID":"ce075a04-45a2-4af0-ba64-2dddd9f9d962","Type":"ContainerStarted","Data":"6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5"}
Oct 30 21:11:59.371599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:59.371557    7037 generic.go:356] "Generic (PLEG): container finished" podID="ce075a04-45a2-4af0-ba64-2dddd9f9d962" containerID="5a6801f42f6be6edae22bd8d7f8e35ec268b04879a0496d3c9016ffb6d3f77ab" exitCode=0
Oct 30 21:11:59.372048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:11:59.371636    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242" event={"ID":"ce075a04-45a2-4af0-ba64-2dddd9f9d962","Type":"ContainerDied","Data":"5a6801f42f6be6edae22bd8d7f8e35ec268b04879a0496d3c9016ffb6d3f77ab"}
Oct 30 21:12:00.521343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.521316    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:12:00.557955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.557921    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\") pod \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") "
Oct 30 21:12:00.558165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.558121    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\" (UniqueName: \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\") pod \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") "
Oct 30 21:12:00.558303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.558173    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\") pod \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\" (UID: \"ce075a04-45a2-4af0-ba64-2dddd9f9d962\") "
Oct 30 21:12:00.558420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.558391    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca" pod "ce075a04-45a2-4af0-ba64-2dddd9f9d962" (UID: "ce075a04-45a2-4af0-ba64-2dddd9f9d962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:00.558767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.558742    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:00.560830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.560799    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca" pod "ce075a04-45a2-4af0-ba64-2dddd9f9d962" (UID: "ce075a04-45a2-4af0-ba64-2dddd9f9d962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:00.560928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.560864    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn" pod "ce075a04-45a2-4af0-ba64-2dddd9f9d962" (UID: "ce075a04-45a2-4af0-ba64-2dddd9f9d962"). InnerVolumeSpecName "kube-api-access-zd5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:00.566977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.566861    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:12:00.567923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.567900    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ce075a04-45a2-4af0-ba64-2dddd9f9d962" containerName="revoke-keys"
Oct 30 21:12:00.567923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.567923    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce075a04-45a2-4af0-ba64-2dddd9f9d962" containerName="revoke-keys"
Oct 30 21:12:00.568291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.568276    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce075a04-45a2-4af0-ba64-2dddd9f9d962" containerName="revoke-keys"
Oct 30 21:12:00.576774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.576734    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.593704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.593667    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:12:00.595752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.595720    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2cda44-openshift-default-dockercfg-4kbl9\""
Oct 30 21:12:00.659650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659637    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659710    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659736    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659753    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659776    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659888    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64wl\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.659977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659931    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.660053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.659995    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.660145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.660127    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd5zn\" (UniqueName: \"kubernetes.io/projected/ce075a04-45a2-4af0-ba64-2dddd9f9d962-kube-api-access-zd5zn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:00.660188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.660149    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce075a04-45a2-4af0-ba64-2dddd9f9d962-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:00.761180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761136    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761223    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761265    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761307    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761337    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761365    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761405    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64wl\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761486    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761591    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761771    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.761860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.761809    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.762120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.762093    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.762217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.762153    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.762571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.762551    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.764156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.764129    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.764359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.764339    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.781364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.781327    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.781547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.781466    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64wl\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:00.909624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:00.909591    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:01.226870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.226841    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:12:01.227287 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:01.227262    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57593a4e_18af_4007_958b_4af655945051.slice/crio-526768a17cc5e1d96618a9ad9202963ea972641378459d96f8e99210ff8d228e WatchSource:0}: Error finding container 526768a17cc5e1d96618a9ad9202963ea972641378459d96f8e99210ff8d228e: Status 404 returned error can't find the container with id 526768a17cc5e1d96618a9ad9202963ea972641378459d96f8e99210ff8d228e
Oct 30 21:12:01.230070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.230036    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:01.230154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.230104    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:01.230154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.230134    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:01.387253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.387209    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" event={"ID":"57593a4e-18af-4007-958b-4af655945051","Type":"ContainerStarted","Data":"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"}
Oct 30 21:12:01.387446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.387285    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" event={"ID":"57593a4e-18af-4007-958b-4af655945051","Type":"ContainerStarted","Data":"526768a17cc5e1d96618a9ad9202963ea972641378459d96f8e99210ff8d228e"}
Oct 30 21:12:01.389258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.389233    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"
Oct 30 21:12:01.389542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.389271    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242" event={"ID":"ce075a04-45a2-4af0-ba64-2dddd9f9d962","Type":"ContainerDied","Data":"6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5"}
Oct 30 21:12:01.389654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.389554    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a76ddd56594ee20df9a4d5679f324703b7158e451e803c76b0348077a951cd5"
Oct 30 21:12:01.433654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.433596    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podStartSLOduration=1.43357675 podStartE2EDuration="1.43357675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:01.432268405 +0000 UTC m=+664.843807846" watchObservedRunningTime="2026-10-30 21:12:01.43357675 +0000 UTC m=+664.845116191"
Oct 30 21:12:01.910243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:01.910204    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:02.912289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:02.912254    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:02.912732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:02.912311    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:03.716770 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:12:03.716719    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:12:03.912477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:03.912430    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:03.913004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:03.912520    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:04.910850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:04.910813    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:04.911039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:04.910871    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:05.910675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:05.910638    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:05.911114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:05.910696    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:06.910339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:06.910300    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:06.910575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:06.910363    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:07.914508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:07.913943    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:12:07.914508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:07.914009    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 21:12:08.303822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.303728    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"]
Oct 30 21:12:08.315828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.315794    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.334897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.334866    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"]
Oct 30 21:12:08.336003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.335971    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:08.336130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.336106    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:12:08.424313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.423721    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.424313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.423767    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.424313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.423843    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxf5\" (UniqueName: \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.525997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.525939    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxf5\" (UniqueName: \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.526204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.526148    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.526204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.526192    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.527447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.527416    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.543466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.543432    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.552514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.551117    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxf5\" (UniqueName: \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\") pod \"maas-api-revoke-sub-286b0cd4-mkgsh\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:08.658539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:08.657723    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:09.086171 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:09.086124    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0813c647_e867_49dc_96af_4df538f358a2.slice/crio-9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73 WatchSource:0}: Error finding container 9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73: Status 404 returned error can't find the container with id 9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73
Oct 30 21:12:09.087421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:09.087381    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"]
Oct 30 21:12:09.522886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:09.522843    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh" event={"ID":"0813c647-e867-49dc-96af-4df538f358a2","Type":"ContainerStarted","Data":"9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73"}
Oct 30 21:12:09.597205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:09.597116    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:09.597798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:09.597543    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:09.601069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:09.601010    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:12:10.530872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:10.530783    7037 generic.go:356] "Generic (PLEG): container finished" podID="0813c647-e867-49dc-96af-4df538f358a2" containerID="1b08cd21ecf3ba7de796f4afa82f0af1d17c0dd63bafa20ae55be39265c8d401" exitCode=0
Oct 30 21:12:10.531233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:10.530874    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh" event={"ID":"0813c647-e867-49dc-96af-4df538f358a2","Type":"ContainerDied","Data":"1b08cd21ecf3ba7de796f4afa82f0af1d17c0dd63bafa20ae55be39265c8d401"}
Oct 30 21:12:11.698107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.698080    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:11.788433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.788347    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\") pod \"0813c647-e867-49dc-96af-4df538f358a2\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") "
Oct 30 21:12:11.788630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.788435    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\" (UniqueName: \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\") pod \"0813c647-e867-49dc-96af-4df538f358a2\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") "
Oct 30 21:12:11.788630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.788474    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\") pod \"0813c647-e867-49dc-96af-4df538f358a2\" (UID: \"0813c647-e867-49dc-96af-4df538f358a2\") "
Oct 30 21:12:11.788915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.788812    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca" pod "0813c647-e867-49dc-96af-4df538f358a2" (UID: "0813c647-e867-49dc-96af-4df538f358a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:11.789190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.789162    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0813c647-e867-49dc-96af-4df538f358a2-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:11.790830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.790802    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5" pod "0813c647-e867-49dc-96af-4df538f358a2" (UID: "0813c647-e867-49dc-96af-4df538f358a2"). InnerVolumeSpecName "kube-api-access-msxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:11.790958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.790933    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca" pod "0813c647-e867-49dc-96af-4df538f358a2" (UID: "0813c647-e867-49dc-96af-4df538f358a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:11.890431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.890397    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxf5\" (UniqueName: \"kubernetes.io/projected/0813c647-e867-49dc-96af-4df538f358a2-kube-api-access-msxf5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:11.890431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.890426    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0813c647-e867-49dc-96af-4df538f358a2-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:11.908527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.905355    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"]
Oct 30 21:12:11.908527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.905978    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0813c647-e867-49dc-96af-4df538f358a2" containerName="revoke-keys"
Oct 30 21:12:11.908527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.906013    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0813c647-e867-49dc-96af-4df538f358a2" containerName="revoke-keys"
Oct 30 21:12:11.908527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.906109    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0813c647-e867-49dc-96af-4df538f358a2" containerName="revoke-keys"
Oct 30 21:12:11.912511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.909710    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:11.932703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.932514    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"]
Oct 30 21:12:11.999775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.999729    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:11.999969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.999874    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxtj\" (UniqueName: \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:11.999969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:11.999911    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.101586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.101547    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.101787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.101638    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxtj\" (UniqueName: \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.101787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.101661    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.102319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.102294    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.104113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.104080    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.120861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.120832    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxtj\" (UniqueName: \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\") pod \"maas-api-revoke-sub-940f171f-4wk5t\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.235057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.235019    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:12.272290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.272255    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"]
Oct 30 21:12:12.277752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.277721    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.325625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.298001    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"]
Oct 30 21:12:12.418801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.418708    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.418949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.418879    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.419010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.418994    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvf9p\" (UniqueName: \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.520217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.520179    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.520388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.520270    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvf9p\" (UniqueName: \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.520388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.520330    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.521045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.521026    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.522667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.522648    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.541521 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:12.534731    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ca0c5a_cea7_40a4_99be_cbc904c3e860.slice/crio-655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047 WatchSource:0}: Error finding container 655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047: Status 404 returned error can't find the container with id 655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047
Oct 30 21:12:12.541521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.536225    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"]
Oct 30 21:12:12.547190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.547163    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvf9p\" (UniqueName: \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\") pod \"maas-api-revoke-sub-4460b9c1-t5vq8\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.549754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.549732    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"
Oct 30 21:12:12.549870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.549762    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh" event={"ID":"0813c647-e867-49dc-96af-4df538f358a2","Type":"ContainerDied","Data":"9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73"}
Oct 30 21:12:12.549870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.549822    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a754006b3b84169778e23eb792ae01be956d1c48e15bc4a412ce2b9111fce73"
Oct 30 21:12:12.551323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.551299    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t" event={"ID":"a7ca0c5a-cea7-40a4-99be-cbc904c3e860","Type":"ContainerStarted","Data":"655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047"}
Oct 30 21:12:12.647014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.646982    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:12.943272 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:12.943237    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d176d15_1437_436d_8cc3_f3bbd3277108.slice/crio-8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f WatchSource:0}: Error finding container 8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f: Status 404 returned error can't find the container with id 8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f
Oct 30 21:12:12.943737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:12.943714    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"]
Oct 30 21:12:13.558635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:13.558533    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8" event={"ID":"3d176d15-1437-436d-8cc3-f3bbd3277108","Type":"ContainerStarted","Data":"bad95b5a7c16d166ccf7cbfc55f2c8d7bcb1b621ee150be7e601d5e04ae721d3"}
Oct 30 21:12:13.558635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:13.558603    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8" event={"ID":"3d176d15-1437-436d-8cc3-f3bbd3277108","Type":"ContainerStarted","Data":"8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f"}
Oct 30 21:12:13.560218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:13.560186    7037 generic.go:356] "Generic (PLEG): container finished" podID="a7ca0c5a-cea7-40a4-99be-cbc904c3e860" containerID="da7df1dd708aec01c2b5057cfbaefc16d1437378af19584dd53d298af87deb87" exitCode=0
Oct 30 21:12:13.560341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:13.560271    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t" event={"ID":"a7ca0c5a-cea7-40a4-99be-cbc904c3e860","Type":"ContainerDied","Data":"da7df1dd708aec01c2b5057cfbaefc16d1437378af19584dd53d298af87deb87"}
Oct 30 21:12:13.600151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:13.600092    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8" podStartSLOduration=1.600077748 podStartE2EDuration="1.600077748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:13.598450126 +0000 UTC m=+677.009989562" watchObservedRunningTime="2026-10-30 21:12:13.600077748 +0000 UTC m=+677.011617212"
Oct 30 21:12:14.568930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.568893    7037 generic.go:356] "Generic (PLEG): container finished" podID="3d176d15-1437-436d-8cc3-f3bbd3277108" containerID="bad95b5a7c16d166ccf7cbfc55f2c8d7bcb1b621ee150be7e601d5e04ae721d3" exitCode=0
Oct 30 21:12:14.569384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.568987    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8" event={"ID":"3d176d15-1437-436d-8cc3-f3bbd3277108","Type":"ContainerDied","Data":"bad95b5a7c16d166ccf7cbfc55f2c8d7bcb1b621ee150be7e601d5e04ae721d3"}
Oct 30 21:12:14.773550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.773518    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:14.849301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.849212    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\") pod \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") "
Oct 30 21:12:14.849462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.849390    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\" (UniqueName: \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\") pod \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") "
Oct 30 21:12:14.849575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.849459    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\") pod \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\" (UID: \"a7ca0c5a-cea7-40a4-99be-cbc904c3e860\") "
Oct 30 21:12:14.849644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.849621    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca" pod "a7ca0c5a-cea7-40a4-99be-cbc904c3e860" (UID: "a7ca0c5a-cea7-40a4-99be-cbc904c3e860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:14.850069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.850033    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:14.851704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.851677    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca" pod "a7ca0c5a-cea7-40a4-99be-cbc904c3e860" (UID: "a7ca0c5a-cea7-40a4-99be-cbc904c3e860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:14.851827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.851772    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj" pod "a7ca0c5a-cea7-40a4-99be-cbc904c3e860" (UID: "a7ca0c5a-cea7-40a4-99be-cbc904c3e860"). InnerVolumeSpecName "kube-api-access-jwxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:14.951542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.951478    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:14.951542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:14.951546    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxtj\" (UniqueName: \"kubernetes.io/projected/a7ca0c5a-cea7-40a4-99be-cbc904c3e860-kube-api-access-jwxtj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:15.156405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.156305    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"]
Oct 30 21:12:15.157016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.156993    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a7ca0c5a-cea7-40a4-99be-cbc904c3e860" containerName="revoke-keys"
Oct 30 21:12:15.157128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.157017    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ca0c5a-cea7-40a4-99be-cbc904c3e860" containerName="revoke-keys"
Oct 30 21:12:15.157220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.157206    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ca0c5a-cea7-40a4-99be-cbc904c3e860" containerName="revoke-keys"
Oct 30 21:12:15.160790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.160768    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.181124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.181092    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"]
Oct 30 21:12:15.255465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.255425    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7ph\" (UniqueName: \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.255639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.255525    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.255679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.255647    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.356825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.356783    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7ph\" (UniqueName: \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.357016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.356854    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.357016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.356968    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.357762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.357732    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.359339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.359313    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.376068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.376041    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7ph\" (UniqueName: \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\") pod \"maas-api-revoke-sub-1f61f33e-dn52g\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.474284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.474203    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:15.577211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.577092    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t" event={"ID":"a7ca0c5a-cea7-40a4-99be-cbc904c3e860","Type":"ContainerDied","Data":"655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047"}
Oct 30 21:12:15.577211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.577171    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655a1a1c43de76ef9fc6a518b10e580c041377c7f7f3674eb696712305e6d047"
Oct 30 21:12:15.577211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.577183    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"
Oct 30 21:12:15.721715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.721685    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:15.789759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.789725    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"]
Oct 30 21:12:15.865952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.864887    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\" (UniqueName: \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\") pod \"3d176d15-1437-436d-8cc3-f3bbd3277108\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") "
Oct 30 21:12:15.865952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.865020    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\") pod \"3d176d15-1437-436d-8cc3-f3bbd3277108\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") "
Oct 30 21:12:15.865952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.865148    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\") pod \"3d176d15-1437-436d-8cc3-f3bbd3277108\" (UID: \"3d176d15-1437-436d-8cc3-f3bbd3277108\") "
Oct 30 21:12:15.866775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.866434    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca" pod "3d176d15-1437-436d-8cc3-f3bbd3277108" (UID: "3d176d15-1437-436d-8cc3-f3bbd3277108"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:15.869727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.869348    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca" pod "3d176d15-1437-436d-8cc3-f3bbd3277108" (UID: "3d176d15-1437-436d-8cc3-f3bbd3277108"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:15.872123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.872089    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p" pod "3d176d15-1437-436d-8cc3-f3bbd3277108" (UID: "3d176d15-1437-436d-8cc3-f3bbd3277108"). InnerVolumeSpecName "kube-api-access-tvf9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:15.966780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.966744    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvf9p\" (UniqueName: \"kubernetes.io/projected/3d176d15-1437-436d-8cc3-f3bbd3277108-kube-api-access-tvf9p\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:15.966780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.966777    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:15.966780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:15.966787    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d176d15-1437-436d-8cc3-f3bbd3277108-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:16.271186 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.271151    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"]
Oct 30 21:12:16.271895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.271873    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3d176d15-1437-436d-8cc3-f3bbd3277108" containerName="revoke-keys"
Oct 30 21:12:16.271974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.271896    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d176d15-1437-436d-8cc3-f3bbd3277108" containerName="revoke-keys"
Oct 30 21:12:16.272097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.272086    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d176d15-1437-436d-8cc3-f3bbd3277108" containerName="revoke-keys"
Oct 30 21:12:16.275912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.275889    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.293196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.293125    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"]
Oct 30 21:12:16.371564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.371527    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.371740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.371670    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.371782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.371753    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xnb\" (UniqueName: \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.472905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.472845    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.473073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.472966    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.473073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.473028    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xnb\" (UniqueName: \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.473787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.473762    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.475230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.475209    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.492954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.492921    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xnb\" (UniqueName: \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\") pod \"maas-api-revoke-sub-ee10ee2a-8qdz9\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.584684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.584655    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"
Oct 30 21:12:16.585154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.584690    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8" event={"ID":"3d176d15-1437-436d-8cc3-f3bbd3277108","Type":"ContainerDied","Data":"8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f"}
Oct 30 21:12:16.585154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.584754    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8195819e8b4a6aafe28d44709dfa47eacfb04576fd3bedd0c2d312840593707f"
Oct 30 21:12:16.586316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.586291    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g" event={"ID":"28888c2d-0013-4038-8fb9-c1f77eb7bf2f","Type":"ContainerStarted","Data":"f707f654805d4bdb12be25086051b47c66018126c15866c30f8cd9914bdf9b4b"}
Oct 30 21:12:16.586435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.586334    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g" event={"ID":"28888c2d-0013-4038-8fb9-c1f77eb7bf2f","Type":"ContainerStarted","Data":"3ba2f3d22292c534ea13682af9a79c2673d176051553da139392ea806479370a"}
Oct 30 21:12:16.589352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.589332    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:16.636393 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.636324    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g" podStartSLOduration=1.6363045710000002 podStartE2EDuration="1.636304571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:16.635216062 +0000 UTC m=+680.046755500" watchObservedRunningTime="2026-10-30 21:12:16.636304571 +0000 UTC m=+680.047844009"
Oct 30 21:12:16.888181 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:16.888142    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23174b83_a192_4abb_aeee_f46bed977cac.slice/crio-0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d WatchSource:0}: Error finding container 0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d: Status 404 returned error can't find the container with id 0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d
Oct 30 21:12:16.888337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:16.888312    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"]
Oct 30 21:12:17.593440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:17.593402    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9" event={"ID":"23174b83-a192-4abb-aeee-f46bed977cac","Type":"ContainerStarted","Data":"6cac683d804c04cf683dc6c6ff1e4d918e9e9a3e2bf23b08bb9162ed80a8c15c"}
Oct 30 21:12:17.593965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:17.593463    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9" event={"ID":"23174b83-a192-4abb-aeee-f46bed977cac","Type":"ContainerStarted","Data":"0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d"}
Oct 30 21:12:17.595051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:17.595014    7037 generic.go:356] "Generic (PLEG): container finished" podID="28888c2d-0013-4038-8fb9-c1f77eb7bf2f" containerID="f707f654805d4bdb12be25086051b47c66018126c15866c30f8cd9914bdf9b4b" exitCode=0
Oct 30 21:12:17.595173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:17.595053    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g" event={"ID":"28888c2d-0013-4038-8fb9-c1f77eb7bf2f","Type":"ContainerDied","Data":"f707f654805d4bdb12be25086051b47c66018126c15866c30f8cd9914bdf9b4b"}
Oct 30 21:12:17.632292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:17.632248    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9" podStartSLOduration=1.632234653 podStartE2EDuration="1.632234653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:17.632006434 +0000 UTC m=+681.043545874" watchObservedRunningTime="2026-10-30 21:12:17.632234653 +0000 UTC m=+681.043774090"
Oct 30 21:12:18.602780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.602746    7037 generic.go:356] "Generic (PLEG): container finished" podID="23174b83-a192-4abb-aeee-f46bed977cac" containerID="6cac683d804c04cf683dc6c6ff1e4d918e9e9a3e2bf23b08bb9162ed80a8c15c" exitCode=0
Oct 30 21:12:18.603256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.602836    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9" event={"ID":"23174b83-a192-4abb-aeee-f46bed977cac","Type":"ContainerDied","Data":"6cac683d804c04cf683dc6c6ff1e4d918e9e9a3e2bf23b08bb9162ed80a8c15c"}
Oct 30 21:12:18.777256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.777228    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:18.901704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.901606    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\" (UniqueName: \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\") pod \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") "
Oct 30 21:12:18.901704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.901677    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\") pod \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") "
Oct 30 21:12:18.901928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.901761    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\") pod \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\" (UID: \"28888c2d-0013-4038-8fb9-c1f77eb7bf2f\") "
Oct 30 21:12:18.902040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.902016    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca" pod "28888c2d-0013-4038-8fb9-c1f77eb7bf2f" (UID: "28888c2d-0013-4038-8fb9-c1f77eb7bf2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:18.902240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.902214    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:18.903907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.903880    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca" pod "28888c2d-0013-4038-8fb9-c1f77eb7bf2f" (UID: "28888c2d-0013-4038-8fb9-c1f77eb7bf2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:18.904009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:18.903914    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph" pod "28888c2d-0013-4038-8fb9-c1f77eb7bf2f" (UID: "28888c2d-0013-4038-8fb9-c1f77eb7bf2f"). InnerVolumeSpecName "kube-api-access-ft7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:19.003407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.003371    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7ph\" (UniqueName: \"kubernetes.io/projected/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-kube-api-access-ft7ph\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:19.003407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.003402    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28888c2d-0013-4038-8fb9-c1f77eb7bf2f-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:19.611940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.611903    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"
Oct 30 21:12:19.611940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.611929    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g" event={"ID":"28888c2d-0013-4038-8fb9-c1f77eb7bf2f","Type":"ContainerDied","Data":"3ba2f3d22292c534ea13682af9a79c2673d176051553da139392ea806479370a"}
Oct 30 21:12:19.612439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.611981    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba2f3d22292c534ea13682af9a79c2673d176051553da139392ea806479370a"
Oct 30 21:12:19.795818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.795795    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:19.933820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.932573    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\" (UniqueName: \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\") pod \"23174b83-a192-4abb-aeee-f46bed977cac\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") "
Oct 30 21:12:19.933820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.932670    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\") pod \"23174b83-a192-4abb-aeee-f46bed977cac\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") "
Oct 30 21:12:19.933820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.932711    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\") pod \"23174b83-a192-4abb-aeee-f46bed977cac\" (UID: \"23174b83-a192-4abb-aeee-f46bed977cac\") "
Oct 30 21:12:19.934828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.934795    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca" pod "23174b83-a192-4abb-aeee-f46bed977cac" (UID: "23174b83-a192-4abb-aeee-f46bed977cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:19.936208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.936168    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca" pod "23174b83-a192-4abb-aeee-f46bed977cac" (UID: "23174b83-a192-4abb-aeee-f46bed977cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:19.938373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:19.938336    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb" pod "23174b83-a192-4abb-aeee-f46bed977cac" (UID: "23174b83-a192-4abb-aeee-f46bed977cac"). InnerVolumeSpecName "kube-api-access-q9xnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:20.036515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.035798    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xnb\" (UniqueName: \"kubernetes.io/projected/23174b83-a192-4abb-aeee-f46bed977cac-kube-api-access-q9xnb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:20.036515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.035836    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:20.036515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.035853    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23174b83-a192-4abb-aeee-f46bed977cac-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:20.620300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.620273    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"
Oct 30 21:12:20.620842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.620261    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9" event={"ID":"23174b83-a192-4abb-aeee-f46bed977cac","Type":"ContainerDied","Data":"0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d"}
Oct 30 21:12:20.620842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:20.620416    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9c3660957dbaa74b8497e3cb01bb5b3bd806fe4e556031135c482abb74583d"
Oct 30 21:12:26.610876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.610835    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"]
Oct 30 21:12:26.611438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611421    7037 cpu_manager.go:414] "containerMap: removing container" podUID="28888c2d-0013-4038-8fb9-c1f77eb7bf2f" containerName="revoke-keys"
Oct 30 21:12:26.611479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611441    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28888c2d-0013-4038-8fb9-c1f77eb7bf2f" containerName="revoke-keys"
Oct 30 21:12:26.611479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611474    7037 cpu_manager.go:414] "containerMap: removing container" podUID="23174b83-a192-4abb-aeee-f46bed977cac" containerName="revoke-keys"
Oct 30 21:12:26.611552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611479    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23174b83-a192-4abb-aeee-f46bed977cac" containerName="revoke-keys"
Oct 30 21:12:26.611602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611592    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="23174b83-a192-4abb-aeee-f46bed977cac" containerName="revoke-keys"
Oct 30 21:12:26.611638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.611609    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="28888c2d-0013-4038-8fb9-c1f77eb7bf2f" containerName="revoke-keys"
Oct 30 21:12:26.615425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.615407    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.633308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.633270    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"]
Oct 30 21:12:26.634043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.634010    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.634206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.634184    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.634277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.634235    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.641917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.641887    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:26.642096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.641979    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:12:26.735074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.735035    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.735263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.735151    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.735263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.735194    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.735911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.735888    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.737630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.737608    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.755299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.755262    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\") pod \"maas-api-revoke-sub-11e70d4f-4sd68\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:26.929309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:26.929225    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:27.226324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.226288    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"]
Oct 30 21:12:27.227145 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:27.227112    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fd9a45_5f13_43ec_a6e3_0e705b9dcfa1.slice/crio-9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae WatchSource:0}: Error finding container 9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae: Status 404 returned error can't find the container with id 9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae
Oct 30 21:12:27.588239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.588208    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"]
Oct 30 21:12:27.592582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.592562    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.613528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.613473    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"]
Oct 30 21:12:27.652432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.652398    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.652616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.652557    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.652655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.652615    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnmr\" (UniqueName: \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.667043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.667002    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68" event={"ID":"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1","Type":"ContainerStarted","Data":"d7dc177b188cd8014f7ef1fbc703270b32003dc598d3a7aa5b8400642c36fcfb"}
Oct 30 21:12:27.667185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.667066    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68" event={"ID":"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1","Type":"ContainerStarted","Data":"9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae"}
Oct 30 21:12:27.707958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.707901    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68" podStartSLOduration=1.707887315 podStartE2EDuration="1.707887315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:27.705298015 +0000 UTC m=+691.116837452" watchObservedRunningTime="2026-10-30 21:12:27.707887315 +0000 UTC m=+691.119426751"
Oct 30 21:12:27.754188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.754153    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.754332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.754239    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnmr\" (UniqueName: \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.754407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.754384    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.755067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.755041    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.756636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.756608    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.773752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.773722    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnmr\" (UniqueName: \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\") pod \"maas-api-revoke-sub-ca3a2156-mxzxx\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:27.911595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:27.911560    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:28.221311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.221278    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"]
Oct 30 21:12:28.222876 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:28.222850    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f9e857_5b49_4a3f_9f54_58f86c34214f.slice/crio-3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86 WatchSource:0}: Error finding container 3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86: Status 404 returned error can't find the container with id 3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86
Oct 30 21:12:28.679459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.679414    7037 generic.go:356] "Generic (PLEG): container finished" podID="33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" containerID="d7dc177b188cd8014f7ef1fbc703270b32003dc598d3a7aa5b8400642c36fcfb" exitCode=0
Oct 30 21:12:28.679972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.679519    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68" event={"ID":"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1","Type":"ContainerDied","Data":"d7dc177b188cd8014f7ef1fbc703270b32003dc598d3a7aa5b8400642c36fcfb"}
Oct 30 21:12:28.681639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.681605    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx" event={"ID":"47f9e857-5b49-4a3f-9f54-58f86c34214f","Type":"ContainerStarted","Data":"722a7747e4cee7c447f8eec793d9c326f6ae1a661365f7815079814fe0dc6eb2"}
Oct 30 21:12:28.681786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.681667    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx" event={"ID":"47f9e857-5b49-4a3f-9f54-58f86c34214f","Type":"ContainerStarted","Data":"3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86"}
Oct 30 21:12:28.762533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:28.762452    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx" podStartSLOduration=1.7624317569999999 podStartE2EDuration="1.762431757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:28.760034814 +0000 UTC m=+692.171574251" watchObservedRunningTime="2026-10-30 21:12:28.762431757 +0000 UTC m=+692.173971195"
Oct 30 21:12:29.637283 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.637245    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"]
Oct 30 21:12:29.647875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.647758    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.655082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.655050    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"]
Oct 30 21:12:29.700180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.700135    7037 generic.go:356] "Generic (PLEG): container finished" podID="47f9e857-5b49-4a3f-9f54-58f86c34214f" containerID="722a7747e4cee7c447f8eec793d9c326f6ae1a661365f7815079814fe0dc6eb2" exitCode=0
Oct 30 21:12:29.700667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.700205    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx" event={"ID":"47f9e857-5b49-4a3f-9f54-58f86c34214f","Type":"ContainerDied","Data":"722a7747e4cee7c447f8eec793d9c326f6ae1a661365f7815079814fe0dc6eb2"}
Oct 30 21:12:29.784828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.784788    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.785030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.784930    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.785030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.784980    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr2q\" (UniqueName: \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.859087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.859049    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:29.886701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.886667    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr2q\" (UniqueName: \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.886901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.886883    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.886975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.886959    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.887733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.887676    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.889381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.889354    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.909341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.909306    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr2q\" (UniqueName: \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\") pod \"maas-api-revoke-sub-74c5e1af-rljms\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.967065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.967030    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:29.987958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.987916    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\") pod \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") "
Oct 30 21:12:29.988103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.987983    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\") pod \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") "
Oct 30 21:12:29.988717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.988689    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca" pod "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" (UID: "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:29.991731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.991701    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw" pod "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" (UID: "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1"). InnerVolumeSpecName "kube-api-access-rwpsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:29.994870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.994843    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca" pod "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" (UID: "33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:29.996576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.988039    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\") pod \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\" (UID: \"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1\") "
Oct 30 21:12:29.997104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.997081    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:29.997104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.997107    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwpsw\" (UniqueName: \"kubernetes.io/projected/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-kube-api-access-rwpsw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:29.997247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:29.997117    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:30.283623 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:30.283587    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4729b2_7727_4c12_8607_f07ae756a7b5.slice/crio-30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4 WatchSource:0}: Error finding container 30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4: Status 404 returned error can't find the container with id 30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4
Oct 30 21:12:30.283750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.283729    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"]
Oct 30 21:12:30.487472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.487438    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:12:30.488017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.488003    7037 cpu_manager.go:414] "containerMap: removing container" podUID="33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" containerName="revoke-keys"
Oct 30 21:12:30.488063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.488019    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" containerName="revoke-keys"
Oct 30 21:12:30.488139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.488130    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" containerName="revoke-keys"
Oct 30 21:12:30.493682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.493654    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.522828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.522790    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-56ac74-openshift-default-dockercfg-gsffb\""
Oct 30 21:12:30.525779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.525755    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:12:30.604423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604432    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604641    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604718    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604751    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604826    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604847    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.604947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.604894    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gm57\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.705701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.705658    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.705759    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.705790    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.705862    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gm57\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.705997    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706026    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706055    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706138    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706172    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706608    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.706683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.706630    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.707076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.707051    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.707260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.707244    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.707470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.707454    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.708559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.708530    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.709155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.709112    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms" event={"ID":"0e4729b2-7727-4c12-8607-f07ae756a7b5","Type":"ContainerStarted","Data":"af473f46255baeaa950f6149bc61a291662d51ca9939635fe2f857531b2c1e38"}
Oct 30 21:12:30.709249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.709169    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms" event={"ID":"0e4729b2-7727-4c12-8607-f07ae756a7b5","Type":"ContainerStarted","Data":"30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4"}
Oct 30 21:12:30.709418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.709392    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.710715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.710699    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"
Oct 30 21:12:30.710806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.710729    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68" event={"ID":"33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1","Type":"ContainerDied","Data":"9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae"}
Oct 30 21:12:30.710806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.710774    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9033f418568c9233861146279678e3da5f29f647b29259a54da0f962f63838ae"
Oct 30 21:12:30.726626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.726593    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.726745 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.726724    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gm57\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.751415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.751363    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms" podStartSLOduration=1.751348548 podStartE2EDuration="1.751348548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:30.750167516 +0000 UTC m=+694.161706955" watchObservedRunningTime="2026-10-30 21:12:30.751348548 +0000 UTC m=+694.162887985"
Oct 30 21:12:30.807680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.807367    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:30.878773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:30.877005    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:31.009235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.009201    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\") pod \"47f9e857-5b49-4a3f-9f54-58f86c34214f\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") "
Oct 30 21:12:31.009428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.009275    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\") pod \"47f9e857-5b49-4a3f-9f54-58f86c34214f\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") "
Oct 30 21:12:31.009428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.009308    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\" (UniqueName: \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\") pod \"47f9e857-5b49-4a3f-9f54-58f86c34214f\" (UID: \"47f9e857-5b49-4a3f-9f54-58f86c34214f\") "
Oct 30 21:12:31.009669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.009637    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca" pod "47f9e857-5b49-4a3f-9f54-58f86c34214f" (UID: "47f9e857-5b49-4a3f-9f54-58f86c34214f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:31.009860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.009839    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:31.011503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.011473    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca" pod "47f9e857-5b49-4a3f-9f54-58f86c34214f" (UID: "47f9e857-5b49-4a3f-9f54-58f86c34214f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:31.011590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.011561    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr" pod "47f9e857-5b49-4a3f-9f54-58f86c34214f" (UID: "47f9e857-5b49-4a3f-9f54-58f86c34214f"). InnerVolumeSpecName "kube-api-access-4mnmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:31.111400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.111362    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f9e857-5b49-4a3f-9f54-58f86c34214f-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:31.111400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.111393    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnmr\" (UniqueName: \"kubernetes.io/projected/47f9e857-5b49-4a3f-9f54-58f86c34214f-kube-api-access-4mnmr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:31.133450 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:31.133414    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70fdb94_df8e_4e9c_aa5d_1c924113345d.slice/crio-dfd1f21c9ae60012d5cd89575de4bb33bda966fa49c28c1666a92c3867553da0 WatchSource:0}: Error finding container dfd1f21c9ae60012d5cd89575de4bb33bda966fa49c28c1666a92c3867553da0: Status 404 returned error can't find the container with id dfd1f21c9ae60012d5cd89575de4bb33bda966fa49c28c1666a92c3867553da0
Oct 30 21:12:31.133450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.133437    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:12:31.135996 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.135959    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:31.136087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.136028    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:31.136087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.136059    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:31.635558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.632288    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"]
Oct 30 21:12:31.635558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.633218    7037 cpu_manager.go:414] "containerMap: removing container" podUID="47f9e857-5b49-4a3f-9f54-58f86c34214f" containerName="revoke-keys"
Oct 30 21:12:31.635558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.633239    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f9e857-5b49-4a3f-9f54-58f86c34214f" containerName="revoke-keys"
Oct 30 21:12:31.635558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.633523    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f9e857-5b49-4a3f-9f54-58f86c34214f" containerName="revoke-keys"
Oct 30 21:12:31.637777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.637747    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.668893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.668854    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"]
Oct 30 21:12:31.721964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.721919    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.722432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.722056    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.722432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.722134    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5bv\" (UniqueName: \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.723107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.723069    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" event={"ID":"f70fdb94-df8e-4e9c-aa5d-1c924113345d","Type":"ContainerStarted","Data":"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"}
Oct 30 21:12:31.723254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.723133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" event={"ID":"f70fdb94-df8e-4e9c-aa5d-1c924113345d","Type":"ContainerStarted","Data":"dfd1f21c9ae60012d5cd89575de4bb33bda966fa49c28c1666a92c3867553da0"}
Oct 30 21:12:31.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.732997    7037 generic.go:356] "Generic (PLEG): container finished" podID="0e4729b2-7727-4c12-8607-f07ae756a7b5" containerID="af473f46255baeaa950f6149bc61a291662d51ca9939635fe2f857531b2c1e38" exitCode=0
Oct 30 21:12:31.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.733218    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms" event={"ID":"0e4729b2-7727-4c12-8607-f07ae756a7b5","Type":"ContainerDied","Data":"af473f46255baeaa950f6149bc61a291662d51ca9939635fe2f857531b2c1e38"}
Oct 30 21:12:31.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.735460    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx" event={"ID":"47f9e857-5b49-4a3f-9f54-58f86c34214f","Type":"ContainerDied","Data":"3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86"}
Oct 30 21:12:31.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.735562    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b74a0e4a580b1e2af04e34e2be3dd8768500a5b7e27140ee799e8a151ca3a86"
Oct 30 21:12:31.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.735651    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"
Oct 30 21:12:31.781108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.781049    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podStartSLOduration=1.7810338300000002 podStartE2EDuration="1.78103383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:31.778670769 +0000 UTC m=+695.190210207" watchObservedRunningTime="2026-10-30 21:12:31.78103383 +0000 UTC m=+695.192573263"
Oct 30 21:12:31.808205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.808171    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:31.823312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.823273    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.823531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.823364    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.823531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.823450    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5bv\" (UniqueName: \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.832513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.831727    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.855517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.852550    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.860663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.860626    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5bv\" (UniqueName: \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\") pod \"maas-api-revoke-sub-51e2cb04-2mk68\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:31.967332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:31.967253    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:32.265646 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:32.265611    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0520c35c_a699_4e8f_9fe0_b1816c41100b.slice/crio-de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a WatchSource:0}: Error finding container de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a: Status 404 returned error can't find the container with id de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a
Oct 30 21:12:32.267846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:32.267820    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"]
Oct 30 21:12:32.754563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:32.754527    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68" event={"ID":"0520c35c-a699-4e8f-9fe0-b1816c41100b","Type":"ContainerStarted","Data":"de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a"}
Oct 30 21:12:32.809844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:32.809804    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:32.810006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:32.809869    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:33.067740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.067709    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:33.164811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.164774    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\") pod \"0e4729b2-7727-4c12-8607-f07ae756a7b5\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") "
Oct 30 21:12:33.165016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.164824    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\") pod \"0e4729b2-7727-4c12-8607-f07ae756a7b5\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") "
Oct 30 21:12:33.165016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.164932    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\" (UniqueName: \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\") pod \"0e4729b2-7727-4c12-8607-f07ae756a7b5\" (UID: \"0e4729b2-7727-4c12-8607-f07ae756a7b5\") "
Oct 30 21:12:33.166004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.165974    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca" pod "0e4729b2-7727-4c12-8607-f07ae756a7b5" (UID: "0e4729b2-7727-4c12-8607-f07ae756a7b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:33.167279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.167258    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q" pod "0e4729b2-7727-4c12-8607-f07ae756a7b5" (UID: "0e4729b2-7727-4c12-8607-f07ae756a7b5"). InnerVolumeSpecName "kube-api-access-hvr2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:33.167830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.167816    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca" pod "0e4729b2-7727-4c12-8607-f07ae756a7b5" (UID: "0e4729b2-7727-4c12-8607-f07ae756a7b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:33.266401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.266363    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr2q\" (UniqueName: \"kubernetes.io/projected/0e4729b2-7727-4c12-8607-f07ae756a7b5-kube-api-access-hvr2q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:33.266401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.266398    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:33.266648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.266412    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4729b2-7727-4c12-8607-f07ae756a7b5-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:33.675339 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:12:33.675306    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:12:33.763784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.763407    7037 generic.go:356] "Generic (PLEG): container finished" podID="0520c35c-a699-4e8f-9fe0-b1816c41100b" containerID="d830563fe20411be26f6eaaba07dc7b0fd03b3b50fa8927e70f3350884b3dc4b" exitCode=0
Oct 30 21:12:33.763784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.763445    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68" event={"ID":"0520c35c-a699-4e8f-9fe0-b1816c41100b","Type":"ContainerDied","Data":"d830563fe20411be26f6eaaba07dc7b0fd03b3b50fa8927e70f3350884b3dc4b"}
Oct 30 21:12:33.766541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.764919    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"
Oct 30 21:12:33.766541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.764957    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms" event={"ID":"0e4729b2-7727-4c12-8607-f07ae756a7b5","Type":"ContainerDied","Data":"30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4"}
Oct 30 21:12:33.766541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.765001    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30342ea33302c7db31e50b3f9d1f2b04a7b74854557b67f24f6414b9d68edda4"
Oct 30 21:12:33.807811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.807712    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:33.807811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:33.807772    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:34.808043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:34.808004    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:34.808470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:34.808071    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:34.964972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:34.964938    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:35.088434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.088397    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\") pod \"0520c35c-a699-4e8f-9fe0-b1816c41100b\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") "
Oct 30 21:12:35.088660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.088537    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\") pod \"0520c35c-a699-4e8f-9fe0-b1816c41100b\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") "
Oct 30 21:12:35.088729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.088689    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\" (UniqueName: \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\") pod \"0520c35c-a699-4e8f-9fe0-b1816c41100b\" (UID: \"0520c35c-a699-4e8f-9fe0-b1816c41100b\") "
Oct 30 21:12:35.088839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.088809    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca" pod "0520c35c-a699-4e8f-9fe0-b1816c41100b" (UID: "0520c35c-a699-4e8f-9fe0-b1816c41100b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:35.089029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.089012    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:35.090688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.090662    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca" pod "0520c35c-a699-4e8f-9fe0-b1816c41100b" (UID: "0520c35c-a699-4e8f-9fe0-b1816c41100b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:35.090856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.090840    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv" pod "0520c35c-a699-4e8f-9fe0-b1816c41100b" (UID: "0520c35c-a699-4e8f-9fe0-b1816c41100b"). InnerVolumeSpecName "kube-api-access-km5bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:35.189947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.189906    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5bv\" (UniqueName: \"kubernetes.io/projected/0520c35c-a699-4e8f-9fe0-b1816c41100b-kube-api-access-km5bv\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:35.189947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.189942    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0520c35c-a699-4e8f-9fe0-b1816c41100b-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:35.779447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.779412    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"
Oct 30 21:12:35.779447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.779442    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68" event={"ID":"0520c35c-a699-4e8f-9fe0-b1816c41100b","Type":"ContainerDied","Data":"de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a"}
Oct 30 21:12:35.779691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.779508    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de219ea4280c1f5e5bd386fa7f80ec75709938e7fcd109feb61de6e8520fde6a"
Oct 30 21:12:35.807925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.807885    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:35.808056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:35.807944    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:36.808511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:36.807804    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:36.808511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:36.807868    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:37.818516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:37.816414    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:12:37.818516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:37.816475    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 21:12:37.955516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:37.954616    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:38.214573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:38.213889    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:38.272921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:38.272574    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:39.852666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.852624    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:12:39.853155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.852692    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:39.854180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.854155    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:39.854540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.854476    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:39.871929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.871897    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:12:39.891778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.891745    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"]
Oct 30 21:12:39.892416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892399    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0520c35c-a699-4e8f-9fe0-b1816c41100b" containerName="revoke-keys"
Oct 30 21:12:39.892456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892418    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0520c35c-a699-4e8f-9fe0-b1816c41100b" containerName="revoke-keys"
Oct 30 21:12:39.892456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892430    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0e4729b2-7727-4c12-8607-f07ae756a7b5" containerName="revoke-keys"
Oct 30 21:12:39.892456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892435    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4729b2-7727-4c12-8607-f07ae756a7b5" containerName="revoke-keys"
Oct 30 21:12:39.892570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892554    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0520c35c-a699-4e8f-9fe0-b1816c41100b" containerName="revoke-keys"
Oct 30 21:12:39.892570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.892563    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4729b2-7727-4c12-8607-f07ae756a7b5" containerName="revoke-keys"
Oct 30 21:12:39.895753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.895737    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:39.901624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.901601    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:39.901746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.901646    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:39.901857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.901837    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lf54\" (UniqueName: \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:39.924656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.924622    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:39.924839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.924749    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:12:39.932309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:39.932271    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"]
Oct 30 21:12:40.003312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.003277    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.003525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.003324    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.003525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.003402    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lf54\" (UniqueName: \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.003993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.003970    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.005919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.005884    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.022982 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.022947    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lf54\" (UniqueName: \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\") pod \"maas-api-revoke-sub-06686839-fwdrn\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.209197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.209114    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:40.516879 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:40.515364    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba1d493_76a0_4bd6_89f4_058f021c506e.slice/crio-5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623 WatchSource:0}: Error finding container 5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623: Status 404 returned error can't find the container with id 5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623
Oct 30 21:12:40.516879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.515942    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"]
Oct 30 21:12:40.884231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:40.884188    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn" event={"ID":"5ba1d493-76a0-4bd6-89f4-058f021c506e","Type":"ContainerStarted","Data":"5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623"}
Oct 30 21:12:41.892531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:41.892478    7037 generic.go:356] "Generic (PLEG): container finished" podID="5ba1d493-76a0-4bd6-89f4-058f021c506e" containerID="fddb844a9d7899a2b901ce580a91e9a9b455b38d25c5860d006d73ce719311f4" exitCode=0
Oct 30 21:12:41.892531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:41.892522    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn" event={"ID":"5ba1d493-76a0-4bd6-89f4-058f021c506e","Type":"ContainerDied","Data":"fddb844a9d7899a2b901ce580a91e9a9b455b38d25c5860d006d73ce719311f4"}
Oct 30 21:12:42.369022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.368976    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"]
Oct 30 21:12:42.373160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.373138    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.391429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.391400    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"]
Oct 30 21:12:42.432798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.432763    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2r5\" (UniqueName: \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.432960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.432813    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.432960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.432907    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.534641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.534607    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2r5\" (UniqueName: \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.534841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.534657    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.534841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.534693    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.535328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.535296    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.536989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.536967    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.553887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.553853    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2r5\" (UniqueName: \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\") pod \"maas-api-revoke-sub-ebcd5343-2g5h6\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.688050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.687957    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:42.990407 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:42.990368    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6357c3_063e_43b7_b923_88876d359f2a.slice/crio-347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74 WatchSource:0}: Error finding container 347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74: Status 404 returned error can't find the container with id 347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74
Oct 30 21:12:42.990906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:42.990578    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"]
Oct 30 21:12:43.038082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.038048    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:43.142783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.142745    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\") pod \"5ba1d493-76a0-4bd6-89f4-058f021c506e\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") "
Oct 30 21:12:43.142961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.142818    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\") pod \"5ba1d493-76a0-4bd6-89f4-058f021c506e\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") "
Oct 30 21:12:43.142961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.142874    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\" (UniqueName: \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\") pod \"5ba1d493-76a0-4bd6-89f4-058f021c506e\" (UID: \"5ba1d493-76a0-4bd6-89f4-058f021c506e\") "
Oct 30 21:12:43.143174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.143149    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca" pod "5ba1d493-76a0-4bd6-89f4-058f021c506e" (UID: "5ba1d493-76a0-4bd6-89f4-058f021c506e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:43.143356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.143334    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:43.144946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.144922    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54" pod "5ba1d493-76a0-4bd6-89f4-058f021c506e" (UID: "5ba1d493-76a0-4bd6-89f4-058f021c506e"). InnerVolumeSpecName "kube-api-access-2lf54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:43.145041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.144928    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca" pod "5ba1d493-76a0-4bd6-89f4-058f021c506e" (UID: "5ba1d493-76a0-4bd6-89f4-058f021c506e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:43.244890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.244796    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba1d493-76a0-4bd6-89f4-058f021c506e-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:43.244890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.244826    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lf54\" (UniqueName: \"kubernetes.io/projected/5ba1d493-76a0-4bd6-89f4-058f021c506e-kube-api-access-2lf54\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:43.908426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.908384    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6" event={"ID":"bd6357c3-063e-43b7-b923-88876d359f2a","Type":"ContainerStarted","Data":"5a81fcb702de281f6176c4d8fc671de86f7b0fe966810be0e2f942e30d96654d"}
Oct 30 21:12:43.908619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.908454    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6" event={"ID":"bd6357c3-063e-43b7-b923-88876d359f2a","Type":"ContainerStarted","Data":"347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74"}
Oct 30 21:12:43.909931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.909909    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"
Oct 30 21:12:43.910023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.909958    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn" event={"ID":"5ba1d493-76a0-4bd6-89f4-058f021c506e","Type":"ContainerDied","Data":"5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623"}
Oct 30 21:12:43.910023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.910006    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b19c9edf6c0f4d66d676ac3e22374f7eaaf63889ba714e00ec23a1ce0051623"
Oct 30 21:12:43.949196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:43.949146    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6" podStartSLOduration=1.949132014 podStartE2EDuration="1.949132014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:43.948029618 +0000 UTC m=+707.359569054" watchObservedRunningTime="2026-10-30 21:12:43.949132014 +0000 UTC m=+707.360671452"
Oct 30 21:12:44.919633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:44.919597    7037 generic.go:356] "Generic (PLEG): container finished" podID="bd6357c3-063e-43b7-b923-88876d359f2a" containerID="5a81fcb702de281f6176c4d8fc671de86f7b0fe966810be0e2f942e30d96654d" exitCode=0
Oct 30 21:12:44.920017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:44.919683    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6" event={"ID":"bd6357c3-063e-43b7-b923-88876d359f2a","Type":"ContainerDied","Data":"5a81fcb702de281f6176c4d8fc671de86f7b0fe966810be0e2f942e30d96654d"}
Oct 30 21:12:45.419270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.419234    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:12:45.420419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.420055    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5ba1d493-76a0-4bd6-89f4-058f021c506e" containerName="revoke-keys"
Oct 30 21:12:45.420419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.420082    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba1d493-76a0-4bd6-89f4-058f021c506e" containerName="revoke-keys"
Oct 30 21:12:45.420419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.420264    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba1d493-76a0-4bd6-89f4-058f021c506e" containerName="revoke-keys"
Oct 30 21:12:45.424354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.424328    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.450166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.450128    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:12:45.472212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472172    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472225    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472309    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tpz\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472387    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472466    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472721    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472744    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472866    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.472953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.472916    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.573979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.573939    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.573992    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574011    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574053    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574080    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574165    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574183    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574217    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tpz\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574249    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574401    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574591    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574736    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.574829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.574802    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.575090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.575068    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.576704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.576683    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.576803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.576745    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.598218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.598189    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.599034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.598989    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tpz\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:45.741120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:45.741038    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:46.058940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.058911    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:46.093669 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:46.093623    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0f73ea_b35a_4aa1_952e_c21b7d09c25d.slice/crio-e1c4b299ab088fb1821cb777cfef17da82603c4fd84bf2d5eff9fc9c935760d6 WatchSource:0}: Error finding container e1c4b299ab088fb1821cb777cfef17da82603c4fd84bf2d5eff9fc9c935760d6: Status 404 returned error can't find the container with id e1c4b299ab088fb1821cb777cfef17da82603c4fd84bf2d5eff9fc9c935760d6
Oct 30 21:12:46.094104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.094077    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\" (UniqueName: \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\") pod \"bd6357c3-063e-43b7-b923-88876d359f2a\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") "
Oct 30 21:12:46.094338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.094314    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\") pod \"bd6357c3-063e-43b7-b923-88876d359f2a\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") "
Oct 30 21:12:46.094443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.094425    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\") pod \"bd6357c3-063e-43b7-b923-88876d359f2a\" (UID: \"bd6357c3-063e-43b7-b923-88876d359f2a\") "
Oct 30 21:12:46.094986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.094806    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca" pod "bd6357c3-063e-43b7-b923-88876d359f2a" (UID: "bd6357c3-063e-43b7-b923-88876d359f2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:46.095107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.094999    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:46.097086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.097051    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca" pod "bd6357c3-063e-43b7-b923-88876d359f2a" (UID: "bd6357c3-063e-43b7-b923-88876d359f2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:46.097086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.097071    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5" pod "bd6357c3-063e-43b7-b923-88876d359f2a" (UID: "bd6357c3-063e-43b7-b923-88876d359f2a"). InnerVolumeSpecName "kube-api-access-2r2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:46.097203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.097123    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:46.097203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.097198    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:46.097266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.097230    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:46.099391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.099369    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:12:46.196437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.196395    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2r5\" (UniqueName: \"kubernetes.io/projected/bd6357c3-063e-43b7-b923-88876d359f2a-kube-api-access-2r2r5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:46.196637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.196443    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6357c3-063e-43b7-b923-88876d359f2a-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:46.955791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.955696    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"
Oct 30 21:12:46.955791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.955728    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6" event={"ID":"bd6357c3-063e-43b7-b923-88876d359f2a","Type":"ContainerDied","Data":"347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74"}
Oct 30 21:12:46.955791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.955785    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347f75a3d552c83d269c09d46e430291bef9d9705dfaff05405eb4ee27197f74"
Oct 30 21:12:46.957846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.957806    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" event={"ID":"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d","Type":"ContainerStarted","Data":"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"}
Oct 30 21:12:46.958083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:46.958061    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" event={"ID":"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d","Type":"ContainerStarted","Data":"e1c4b299ab088fb1821cb777cfef17da82603c4fd84bf2d5eff9fc9c935760d6"}
Oct 30 21:12:47.053551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:47.051436    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podStartSLOduration=2.051416399 podStartE2EDuration="2.051416399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:47.018413972 +0000 UTC m=+710.429953408" watchObservedRunningTime="2026-10-30 21:12:47.051416399 +0000 UTC m=+710.462955836"
Oct 30 21:12:47.751514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:47.749709    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:47.751514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:47.749894    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:47.751514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:47.749959    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:48.141514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.135707    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:12:48.141514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.136754    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bd6357c3-063e-43b7-b923-88876d359f2a" containerName="revoke-keys"
Oct 30 21:12:48.141514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.136780    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6357c3-063e-43b7-b923-88876d359f2a" containerName="revoke-keys"
Oct 30 21:12:48.141514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.136932    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6357c3-063e-43b7-b923-88876d359f2a" containerName="revoke-keys"
Oct 30 21:12:48.175596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.170848    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.175596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.171709    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:12:48.263420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.263321    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.263701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.263675    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.263897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.263879    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264038    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264192    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264314    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264474    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jcs\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264621    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.264776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.264759    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377343    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377447    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377528    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377598    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377631    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377705    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jcs\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.377785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377741    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.378058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377799    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.378058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.377866    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.453845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.433153    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.453845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.433449    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.453845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.433760    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.453845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.435298    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.453845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.435902    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.510977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.492590    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.511892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.511823    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.546950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.532908    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.546950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.534239    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jcs\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-vp9n6\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.609141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.608750    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:48.756515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.751816    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:48.756515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:48.751875    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:49.218025 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:49.217991    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e82098_0ef4_4db4_bf3d_fbf5a3e59bf0.slice/crio-8ab02323b5518b13ade98e6de6b1ec594a6331bd760db3bc846fe85a47141a3f WatchSource:0}: Error finding container 8ab02323b5518b13ade98e6de6b1ec594a6331bd760db3bc846fe85a47141a3f: Status 404 returned error can't find the container with id 8ab02323b5518b13ade98e6de6b1ec594a6331bd760db3bc846fe85a47141a3f
Oct 30 21:12:49.218592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.218386    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:12:49.220577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.220423    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:49.220667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.220616    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:49.220667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.220648    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:12:49.746061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.746016    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:49.746323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:49.746293    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:50.110721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.110677    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" event={"ID":"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0","Type":"ContainerStarted","Data":"48dd3c538e283dc741e32efd57bf2a22ce5bf61e8a1bf4dd117c61a857436491"}
Oct 30 21:12:50.111251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.110751    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" event={"ID":"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0","Type":"ContainerStarted","Data":"8ab02323b5518b13ade98e6de6b1ec594a6331bd760db3bc846fe85a47141a3f"}
Oct 30 21:12:50.175889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.175821    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podStartSLOduration=2.1758019060000002 podStartE2EDuration="2.175801906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:50.172380015 +0000 UTC m=+713.583919465" watchObservedRunningTime="2026-10-30 21:12:50.175801906 +0000 UTC m=+713.587341344"
Oct 30 21:12:50.610141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.610094    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:50.611726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.611702    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:50.611817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.611751    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:50.757515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.756663    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:50.757515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:50.756729    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:51.609774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:51.609740    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:51.610151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:51.609789    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:51.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:51.742920    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:51.746516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:51.742985    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:52.610599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:52.610557    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:52.611105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:52.610643    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:52.760930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:52.760888    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:12:52.761242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:52.761209    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:12:53.062754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.062656    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"]
Oct 30 21:12:53.069483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.069447    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.084305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.084272    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"]
Oct 30 21:12:53.084547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.084528    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:53.093649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.093614    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:12:53.117551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.117472    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.117889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.117863    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.118049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.118032    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pz4\" (UniqueName: \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.219746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.219685    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.219960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.219804    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.219960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.219845    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pz4\" (UniqueName: \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.220362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.220332    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.222164 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.222140    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.255917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.255877    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pz4\" (UniqueName: \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\") pod \"maas-api-revoke-sub-9e5894c8-twx76\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.388755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.388725    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:53.610075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.610036    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:53.610254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.610103    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:53.713160 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:12:53.713133    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab1522b_86d3_420a_a9ea_4ac3213e507a.slice/crio-5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3 WatchSource:0}: Error finding container 5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3: Status 404 returned error can't find the container with id 5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3
Oct 30 21:12:53.713523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:53.713461    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"]
Oct 30 21:12:54.102437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.102407    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:54.102809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.102786    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:54.103749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.103720    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:12:54.174072 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.174032    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76" event={"ID":"4ab1522b-86d3-420a-a9ea-4ac3213e507a","Type":"ContainerStarted","Data":"e15191ac87aae42d0b230b69991508c0d9db117df92f66071f2a7c5e8c9c3fe6"}
Oct 30 21:12:54.174239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.174092    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76" event={"ID":"4ab1522b-86d3-420a-a9ea-4ac3213e507a","Type":"ContainerStarted","Data":"5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3"}
Oct 30 21:12:54.300117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.300065    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76" podStartSLOduration=1.300050244 podStartE2EDuration="1.300050244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:54.296939869 +0000 UTC m=+717.708479307" watchObservedRunningTime="2026-10-30 21:12:54.300050244 +0000 UTC m=+717.711589681"
Oct 30 21:12:54.610045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.610005    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:54.610223 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:54.610073    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:55.208572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:55.206594    7037 generic.go:356] "Generic (PLEG): container finished" podID="4ab1522b-86d3-420a-a9ea-4ac3213e507a" containerID="e15191ac87aae42d0b230b69991508c0d9db117df92f66071f2a7c5e8c9c3fe6" exitCode=0
Oct 30 21:12:55.208572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:55.208276    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76" event={"ID":"4ab1522b-86d3-420a-a9ea-4ac3213e507a","Type":"ContainerDied","Data":"e15191ac87aae42d0b230b69991508c0d9db117df92f66071f2a7c5e8c9c3fe6"}
Oct 30 21:12:55.610007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:55.609946    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:55.610191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:55.610037    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:56.384256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.384224    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:56.455888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.455844    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\") pod \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") "
Oct 30 21:12:56.456089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.455960    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\" (UniqueName: \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\") pod \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") "
Oct 30 21:12:56.456089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.456001    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\") pod \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\" (UID: \"4ab1522b-86d3-420a-a9ea-4ac3213e507a\") "
Oct 30 21:12:56.456530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.456450    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca" pod "4ab1522b-86d3-420a-a9ea-4ac3213e507a" (UID: "4ab1522b-86d3-420a-a9ea-4ac3213e507a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:56.456661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.456642    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:56.458747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.458712    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4" pod "4ab1522b-86d3-420a-a9ea-4ac3213e507a" (UID: "4ab1522b-86d3-420a-a9ea-4ac3213e507a"). InnerVolumeSpecName "kube-api-access-g5pz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:56.458863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.458718    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca" pod "4ab1522b-86d3-420a-a9ea-4ac3213e507a" (UID: "4ab1522b-86d3-420a-a9ea-4ac3213e507a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:56.558194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.558152    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab1522b-86d3-420a-a9ea-4ac3213e507a-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:56.558194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.558193    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pz4\" (UniqueName: \"kubernetes.io/projected/4ab1522b-86d3-420a-a9ea-4ac3213e507a-kube-api-access-g5pz4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:12:56.609545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.609504    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-vp9n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:56.609731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:56.609564    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:12:57.221360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:57.221329    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"
Oct 30 21:12:57.221600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:57.221361    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76" event={"ID":"4ab1522b-86d3-420a-a9ea-4ac3213e507a","Type":"ContainerDied","Data":"5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3"}
Oct 30 21:12:57.221600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:57.221417    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5476bc91179b06bd776ad71d1f1e8412aed66d1c36e87a93e050289a425623a3"
Oct 30 21:12:58.158274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:58.158241    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:58.158806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:58.158540    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:12:58.159404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:12:58.159386    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:13:00.255258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.255226    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:13:00.255889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.255825    7037 cpu_manager.go:414] "containerMap: removing container" podUID="4ab1522b-86d3-420a-a9ea-4ac3213e507a" containerName="revoke-keys"
Oct 30 21:13:00.255889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.255839    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab1522b-86d3-420a-a9ea-4ac3213e507a" containerName="revoke-keys"
Oct 30 21:13:00.255999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.255927    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab1522b-86d3-420a-a9ea-4ac3213e507a" containerName="revoke-keys"
Oct 30 21:13:00.260366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.260332    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.310069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.310028    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:13:00.310830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.310794    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.310981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.310857    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.310993    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311107    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92j2\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311186    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311226    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311302    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311349    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.311439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.311408    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.412976    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92j2\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413027    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413050    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413097    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413113    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413157    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413192    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413229    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413312    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413533    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413635    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413750    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413777    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.413948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.413925    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.415654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.415626    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.415794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.415775    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.432974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.432939    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.433171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.433146    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92j2\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.524905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.524820    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"]
Oct 30 21:13:00.530228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.530204    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.554971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.554929    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:00.555151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.555037    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:13:00.558556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.558526    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"]
Oct 30 21:13:00.601667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.601622    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:00.615435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.615404    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvng9\" (UniqueName: \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.615574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.615444    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.615782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.615756    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.717691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.717652    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvng9\" (UniqueName: \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.717691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.717696    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.717953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.717798    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.718871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.718840    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.720101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.720079    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.739151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.739125    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvng9\" (UniqueName: \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\") pod \"maas-api-revoke-sub-83d73c08-8xrvw\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.846500 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.846469    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:00.916148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.916112    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:13:00.916751 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:00.916717    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30262c1c_c6bc_49a7_b1cc_238910051a8b.slice/crio-5ea480b32993b1502585bd33cb71502c296b4d45bb17ee939cecd7c1f88f433a WatchSource:0}: Error finding container 5ea480b32993b1502585bd33cb71502c296b4d45bb17ee939cecd7c1f88f433a: Status 404 returned error can't find the container with id 5ea480b32993b1502585bd33cb71502c296b4d45bb17ee939cecd7c1f88f433a
Oct 30 21:13:00.919604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.919559    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:00.919714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.919652    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:00.919714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:00.919693    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:01.144332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.144230    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"]
Oct 30 21:13:01.256455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.256177    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" event={"ID":"30262c1c-c6bc-49a7-b1cc-238910051a8b","Type":"ContainerStarted","Data":"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"}
Oct 30 21:13:01.256455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.256260    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" event={"ID":"30262c1c-c6bc-49a7-b1cc-238910051a8b","Type":"ContainerStarted","Data":"5ea480b32993b1502585bd33cb71502c296b4d45bb17ee939cecd7c1f88f433a"}
Oct 30 21:13:01.257368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.257337    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:13:01.263861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.263833    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw" event={"ID":"da452ee6-fd5c-4938-a3b0-78924406360e","Type":"ContainerStarted","Data":"2f8c409e08a8f440c737e37be765d100102784942cc37aa2df89423fdae3f76d"}
Oct 30 21:13:01.263994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.263930    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.281309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.281261    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"distinct-w0-58fea9-kserve-self-signed-certs\""
Oct 30 21:13:01.285992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.285959    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:13:01.303604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.303552    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podStartSLOduration=1.303537138 podStartE2EDuration="1.303537138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:01.302232616 +0000 UTC m=+724.713772054" watchObservedRunningTime="2026-10-30 21:13:01.303537138 +0000 UTC m=+724.715076574"
Oct 30 21:13:01.327902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.327856    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.328063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.327978    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.328366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.328344    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.328479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.328453    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.328578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.328544    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.328648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.328579    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8v4\" (UniqueName: \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.429997    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430055    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430093    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430172    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430212    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430251    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8v4\" (UniqueName: \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430459    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.430733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.430703    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.432817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.432796    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.433017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.432996    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.451610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.451579    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8v4\" (UniqueName: \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\") pod \"distinct-w0-58fea9-kserve-77d44f99d7-xqsnb\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.586472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.586429    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:01.602396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.602368    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:01.603961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.603929    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:01.604085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.603991    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:01.890655 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:01.890622    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod141a6dca_9c9e_44eb_9b63_58bf541f80ba.slice/crio-28981e0087da4650a25354fc7f5d69a8bb333a35b3a550905333b899f299d594 WatchSource:0}: Error finding container 28981e0087da4650a25354fc7f5d69a8bb333a35b3a550905333b899f299d594: Status 404 returned error can't find the container with id 28981e0087da4650a25354fc7f5d69a8bb333a35b3a550905333b899f299d594
Oct 30 21:13:01.891012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:01.890989    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:13:02.267549 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.267435    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" event={"ID":"141a6dca-9c9e-44eb-9b63-58bf541f80ba","Type":"ContainerStarted","Data":"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"}
Oct 30 21:13:02.267549 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.267515    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" event={"ID":"141a6dca-9c9e-44eb-9b63-58bf541f80ba","Type":"ContainerStarted","Data":"28981e0087da4650a25354fc7f5d69a8bb333a35b3a550905333b899f299d594"}
Oct 30 21:13:02.269149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.269125    7037 generic.go:356] "Generic (PLEG): container finished" podID="da452ee6-fd5c-4938-a3b0-78924406360e" containerID="f99cc3af037247eb6de8e13c73c2b615991398d11ac553dd6eae54eba158c28b" exitCode=0
Oct 30 21:13:02.269252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.269206    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw" event={"ID":"da452ee6-fd5c-4938-a3b0-78924406360e","Type":"ContainerDied","Data":"f99cc3af037247eb6de8e13c73c2b615991398d11ac553dd6eae54eba158c28b"}
Oct 30 21:13:02.309245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.309196    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podStartSLOduration=1.309182753 podStartE2EDuration="1.309182753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:02.307668142 +0000 UTC m=+725.719207579" watchObservedRunningTime="2026-10-30 21:13:02.309182753 +0000 UTC m=+725.720722238"
Oct 30 21:13:02.586694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.586656    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:02.602060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.602027    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:02.602172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:02.602087    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:03.169579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.169536    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:13:03.180161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.180124    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.194323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.194275    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:13:03.276804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276557    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.276804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276673    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.276804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276751    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.276804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276782    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276834    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276874    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276907    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.276931    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfpfd\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.277185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.277003    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378103    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378161    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378202    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378243    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378330    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378416    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378455    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfpfd\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378661    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.378759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.378711    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.379756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.379100    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.379756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.379280    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.379756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.379298    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.379756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.379604    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.380179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.380162    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.382329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.381780    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.382329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.382033    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.399024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.398989    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfpfd\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.400323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.400269    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7kdrm\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.453969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.453894    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:03.495778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.495711    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:03.581077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.581036    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\") pod \"da452ee6-fd5c-4938-a3b0-78924406360e\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") "
Oct 30 21:13:03.581274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.581087    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\") pod \"da452ee6-fd5c-4938-a3b0-78924406360e\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") "
Oct 30 21:13:03.581274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.581195    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\" (UniqueName: \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\") pod \"da452ee6-fd5c-4938-a3b0-78924406360e\" (UID: \"da452ee6-fd5c-4938-a3b0-78924406360e\") "
Oct 30 21:13:03.581949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.581911    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca" pod "da452ee6-fd5c-4938-a3b0-78924406360e" (UID: "da452ee6-fd5c-4938-a3b0-78924406360e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:03.583991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.583955    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9" pod "da452ee6-fd5c-4938-a3b0-78924406360e" (UID: "da452ee6-fd5c-4938-a3b0-78924406360e"). InnerVolumeSpecName "kube-api-access-vvng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:03.584629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.584608    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca" pod "da452ee6-fd5c-4938-a3b0-78924406360e" (UID: "da452ee6-fd5c-4938-a3b0-78924406360e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:03.602704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.602135    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:03.602704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.602193    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:03.683636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.683604    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvng9\" (UniqueName: \"kubernetes.io/projected/da452ee6-fd5c-4938-a3b0-78924406360e-kube-api-access-vvng9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:03.683636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.683634    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:03.683853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.683648    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da452ee6-fd5c-4938-a3b0-78924406360e-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:03.684339 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:13:03.684301    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:13:03.855523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.855475    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:13:03.856903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.856866    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:03.856992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.856933    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:03.856992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:03.856964    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:04.289467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.289366    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" event={"ID":"f995af46-5f95-4be6-85c1-4b468404c1b0","Type":"ContainerStarted","Data":"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"}
Oct 30 21:13:04.289467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.289441    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" event={"ID":"f995af46-5f95-4be6-85c1-4b468404c1b0","Type":"ContainerStarted","Data":"203d114ff63d4e2febb646db751e34fac969dbd275c38dc6b4b72467736ce0e9"}
Oct 30 21:13:04.293885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.293846    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"
Oct 30 21:13:04.294177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.294150    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw" event={"ID":"da452ee6-fd5c-4938-a3b0-78924406360e","Type":"ContainerDied","Data":"2f8c409e08a8f440c737e37be765d100102784942cc37aa2df89423fdae3f76d"}
Oct 30 21:13:04.294224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.294203    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f8c409e08a8f440c737e37be765d100102784942cc37aa2df89423fdae3f76d"
Oct 30 21:13:04.496477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.496441    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:04.498100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.498053    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:04.498100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.498092    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:04.513261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.513193    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podStartSLOduration=1.513173225 podStartE2EDuration="1.513173225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:04.355633707 +0000 UTC m=+727.767173146" watchObservedRunningTime="2026-10-30 21:13:04.513173225 +0000 UTC m=+727.924712661"
Oct 30 21:13:04.602000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.601965    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:04.602185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:04.602022    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:05.496593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:05.496557    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:05.497015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:05.496615    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:05.604086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:05.604051    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:05.604374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:05.604328    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:06.497514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:06.496963    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:06.497514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:06.497048    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:06.602867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:06.602825    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:06.603155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:06.603126    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:07.120520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.104633    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 30 21:13:07.251717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.251653    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:07.496708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.496611    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:07.496708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.496678    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:07.602569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.602533    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:13:07.602946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:07.602587    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:13:08.496854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:08.496818    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:08.497046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:08.496878    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:08.606288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:08.606256    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:08.606774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:08.606751    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:08.607919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:08.607897    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:13:09.496563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:09.496530    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:09.496732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:09.496583    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:10.496428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:10.496383    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:10.496852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:10.496454    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:11.497129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:11.497089    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7kdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:13:11.497683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:11.497156    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 21:13:11.587197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:11.587161    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:11.600096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:11.600068    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:12.489712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:12.489676    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:13:12.904647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:12.904610    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:12.905153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:12.904964    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:12.905962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:12.905940    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:13:15.570739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.570706    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:13:15.571301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.571287    7037 cpu_manager.go:414] "containerMap: removing container" podUID="da452ee6-fd5c-4938-a3b0-78924406360e" containerName="revoke-keys"
Oct 30 21:13:15.571340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.571302    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da452ee6-fd5c-4938-a3b0-78924406360e" containerName="revoke-keys"
Oct 30 21:13:15.571408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.571399    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="da452ee6-fd5c-4938-a3b0-78924406360e" containerName="revoke-keys"
Oct 30 21:13:15.575478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.575457    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.595313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.595288    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-839163-openshift-default-dockercfg-lpmcf\""
Oct 30 21:13:15.612509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.612362    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:13:15.684352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684313    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684477    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684565    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684643    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684801    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684830    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.684931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.684915    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.685063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.685038    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.685193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.685173    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rmnd\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786267    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786330    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786373    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786475    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rmnd\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786587    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786656    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786787    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786857    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786938    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.786988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.786957    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.787105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.787081    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.787189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.787168    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.787189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.787179    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.787376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.787358    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.788955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.788930    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.789231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.789214    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.806163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.806128    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.811516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.810421    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rmnd\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:15.896013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:15.895918    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:16.209668 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:16.209635    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782e5973_dc08_4e8a_916a_b99deef909c0.slice/crio-a0e0fddd6b35bb7bc7e4ccb47f863e67219359d1491d16de7e4cfd3a62f27dfa WatchSource:0}: Error finding container a0e0fddd6b35bb7bc7e4ccb47f863e67219359d1491d16de7e4cfd3a62f27dfa: Status 404 returned error can't find the container with id a0e0fddd6b35bb7bc7e4ccb47f863e67219359d1491d16de7e4cfd3a62f27dfa
Oct 30 21:13:16.210976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.210944    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:13:16.212749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.212703    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:16.212858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.212794    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:16.212858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.212838    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:16.507689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.507589    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" event={"ID":"782e5973-dc08-4e8a-916a-b99deef909c0","Type":"ContainerStarted","Data":"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"}
Oct 30 21:13:16.507689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.507657    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" event={"ID":"782e5973-dc08-4e8a-916a-b99deef909c0","Type":"ContainerStarted","Data":"a0e0fddd6b35bb7bc7e4ccb47f863e67219359d1491d16de7e4cfd3a62f27dfa"}
Oct 30 21:13:16.576573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.576507    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podStartSLOduration=1.5764751540000002 podStartE2EDuration="1.576475154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:16.561746946 +0000 UTC m=+739.973286384" watchObservedRunningTime="2026-10-30 21:13:16.576475154 +0000 UTC m=+739.988014590"
Oct 30 21:13:16.896951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:16.896915    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:17.899511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:17.899045    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:17.899511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:17.899138    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:18.407674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.407615    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"]
Oct 30 21:13:18.421150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.421117    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.498515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.483762    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"]
Oct 30 21:13:18.535511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.511718    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:13:18.562516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.561746    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:18.703574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.697237    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.703574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.697315    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.703574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.697354    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.826467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.825558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.826467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.825665    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.826467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.825711    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.852736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.842746    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:18.871515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:18.855448    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:19.095517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.078851    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:19.095517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.078940    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:19.095517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.080138    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\") pod \"maas-api-revoke-sub-b03b7a9c-fnmx6\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:19.143982 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.143932    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:19.541697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.541648    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:19.916059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.916024    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:19.916738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:19.916705    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:20.102649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.102614    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"]
Oct 30 21:13:20.896559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.896512    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:20.896762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.896581    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:20.951880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.951831    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6" event={"ID":"62f7bffa-842c-400a-a270-d384b12cf1d1","Type":"ContainerStarted","Data":"9777c4d57d6cab4ba6b4af0d45e372671f5198982157d8485756147daaaf0c5d"}
Oct 30 21:13:20.952065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.951908    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6" event={"ID":"62f7bffa-842c-400a-a270-d384b12cf1d1","Type":"ContainerStarted","Data":"517fded330fdefb06cac44913cd2601d912e2e75c7d58ceba31b7ae689d4615a"}
Oct 30 21:13:20.994809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:20.994750    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6" podStartSLOduration=2.994736531 podStartE2EDuration="2.994736531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:20.993872969 +0000 UTC m=+744.405412406" watchObservedRunningTime="2026-10-30 21:13:20.994736531 +0000 UTC m=+744.406275972"
Oct 30 21:13:21.559852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.559812    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:13:21.565194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.565168    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.584688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.584608    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:13:21.591338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.591308    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"distinct-w4-839163-kserve-self-signed-certs\""
Oct 30 21:13:21.620587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.620547    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.620743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.620671    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.620806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.620741    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.620929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.620906    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227tk\" (UniqueName: \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.620983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.620953    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.621079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.621058    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722574    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722659    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722743    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722864    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227tk\" (UniqueName: \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722900    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.722953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.722947    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.723111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.723042    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.723171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.723134    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.723231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.723177    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.753904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.753870    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.754628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.754592    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.757052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.757026    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227tk\" (UniqueName: \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\") pod \"distinct-w4-839163-kserve-6c4bbcd677-549nk\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") " pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.880762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.880658    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:21.896857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.896822    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:21.896995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.896880    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:21.961507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.961464    7037 generic.go:356] "Generic (PLEG): container finished" podID="62f7bffa-842c-400a-a270-d384b12cf1d1" containerID="9777c4d57d6cab4ba6b4af0d45e372671f5198982157d8485756147daaaf0c5d" exitCode=0
Oct 30 21:13:21.961696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:21.961618    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6" event={"ID":"62f7bffa-842c-400a-a270-d384b12cf1d1","Type":"ContainerDied","Data":"9777c4d57d6cab4ba6b4af0d45e372671f5198982157d8485756147daaaf0c5d"}
Oct 30 21:13:22.213903 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:22.213864    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a7d957_d295_4ee4_9588_57819e924eeb.slice/crio-66bab6ece5151b14e50fc275f24cf31c4041e2d7e60a10056a3b44f3acf957b8 WatchSource:0}: Error finding container 66bab6ece5151b14e50fc275f24cf31c4041e2d7e60a10056a3b44f3acf957b8: Status 404 returned error can't find the container with id 66bab6ece5151b14e50fc275f24cf31c4041e2d7e60a10056a3b44f3acf957b8
Oct 30 21:13:22.216563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:22.216527    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:13:22.896933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:22.896896    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:13:22.897310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:22.896952    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:13:22.971829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:22.971786    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" event={"ID":"a0a7d957-d295-4ee4-9588-57819e924eeb","Type":"ContainerStarted","Data":"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"}
Oct 30 21:13:22.972048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:22.971841    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" event={"ID":"a0a7d957-d295-4ee4-9588-57819e924eeb","Type":"ContainerStarted","Data":"66bab6ece5151b14e50fc275f24cf31c4041e2d7e60a10056a3b44f3acf957b8"}
Oct 30 21:13:23.016623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.016562    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podStartSLOduration=2.01654645 podStartE2EDuration="2.01654645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:23.012599911 +0000 UTC m=+746.424139348" watchObservedRunningTime="2026-10-30 21:13:23.01654645 +0000 UTC m=+746.428085886"
Oct 30 21:13:23.150275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.150149    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:23.271723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.271686    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\") pod \"62f7bffa-842c-400a-a270-d384b12cf1d1\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") "
Oct 30 21:13:23.271921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.271805    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\") pod \"62f7bffa-842c-400a-a270-d384b12cf1d1\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") "
Oct 30 21:13:23.271921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.271838    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\") pod \"62f7bffa-842c-400a-a270-d384b12cf1d1\" (UID: \"62f7bffa-842c-400a-a270-d384b12cf1d1\") "
Oct 30 21:13:23.272213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.272177    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca" pod "62f7bffa-842c-400a-a270-d384b12cf1d1" (UID: "62f7bffa-842c-400a-a270-d384b12cf1d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:23.272377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.272349    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:23.274155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.274117    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca" pod "62f7bffa-842c-400a-a270-d384b12cf1d1" (UID: "62f7bffa-842c-400a-a270-d384b12cf1d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:23.274320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.274300    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd" pod "62f7bffa-842c-400a-a270-d384b12cf1d1" (UID: "62f7bffa-842c-400a-a270-d384b12cf1d1"). InnerVolumeSpecName "kube-api-access-h7djd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:23.373518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.373454    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f7bffa-842c-400a-a270-d384b12cf1d1-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:23.373518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.373512    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7djd\" (UniqueName: \"kubernetes.io/projected/62f7bffa-842c-400a-a270-d384b12cf1d1-kube-api-access-h7djd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:13:23.881034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.880995    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:23.901979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.901948    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:23.902391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.902370    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:23.903256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.903235    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:13:23.980033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.979995    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"
Oct 30 21:13:23.980033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.980025    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6" event={"ID":"62f7bffa-842c-400a-a270-d384b12cf1d1","Type":"ContainerDied","Data":"517fded330fdefb06cac44913cd2601d912e2e75c7d58ceba31b7ae689d4615a"}
Oct 30 21:13:23.980269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:23.980072    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517fded330fdefb06cac44913cd2601d912e2e75c7d58ceba31b7ae689d4615a"
Oct 30 21:13:30.574523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.574472    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:13:30.575087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.575073    7037 cpu_manager.go:414] "containerMap: removing container" podUID="62f7bffa-842c-400a-a270-d384b12cf1d1" containerName="revoke-keys"
Oct 30 21:13:30.575126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.575088    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f7bffa-842c-400a-a270-d384b12cf1d1" containerName="revoke-keys"
Oct 30 21:13:30.575202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.575192    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f7bffa-842c-400a-a270-d384b12cf1d1" containerName="revoke-keys"
Oct 30 21:13:30.578772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.578752    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.600283 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.600251    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:13:30.772582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772546    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.772582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772586    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.772806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772670    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.772806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772715    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.772904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772858    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.772980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.772965    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.773044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.773030    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.773110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.773098    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.773183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.773162    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzc4\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874469    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874557    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzc4\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874618    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874645    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874688    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.874883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874797    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.874911    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875073    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875139    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875197    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875208    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875433    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875730    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.875898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.875875    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.877121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.877094    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.877346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.877325    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.896275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.896237    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:30.901130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:30.901093    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzc4\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:31.192569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.192426    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:31.208323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.208299    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:13:31.213862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.213838    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.248387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.248356    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:13:31.280310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280269    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280382    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280411    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280465    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280596    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfqj\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280624    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280689    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280725    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.280912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.280768    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.381934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.381899    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.381954    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.381995    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382096    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382273    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382303    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382367    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382432    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfqj\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382473    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382691    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382711    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382865    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.382947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382872    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.383003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.382946    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.384435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.384413    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.384566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.384550    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.401853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.401820    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.402160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.402136    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfqj\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.517073 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:31.517042    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e23218_42fc_48c9_8b42_b3e56ce080f9.slice/crio-57589285c0d92a7ef6dc1af4d2fc1fb9891fb56390f248d0ff0c618914d62c5b WatchSource:0}: Error finding container 57589285c0d92a7ef6dc1af4d2fc1fb9891fb56390f248d0ff0c618914d62c5b: Status 404 returned error can't find the container with id 57589285c0d92a7ef6dc1af4d2fc1fb9891fb56390f248d0ff0c618914d62c5b
Oct 30 21:13:31.518733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.518707    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:13:31.519994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.519958    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.520066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.520023    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.520066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.520051    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.545818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.545789    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:31.858659 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:31.858619    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d1de9d_18ae_45ff_bd47_c3def11ef235.slice/crio-8ceff4daf793551592086363ce30dac40963f69c85ec91ad67dc5505b664979d WatchSource:0}: Error finding container 8ceff4daf793551592086363ce30dac40963f69c85ec91ad67dc5505b664979d: Status 404 returned error can't find the container with id 8ceff4daf793551592086363ce30dac40963f69c85ec91ad67dc5505b664979d
Oct 30 21:13:31.860704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.860675    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:13:31.861867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.861828    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.861993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.861898    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.861993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.861928    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:13:31.880937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.880912    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:31.894486 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:31.894456    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:32.059191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.059140    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" event={"ID":"61d1de9d-18ae-45ff-bd47-c3def11ef235","Type":"ContainerStarted","Data":"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"}
Oct 30 21:13:32.059392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.059211    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" event={"ID":"61d1de9d-18ae-45ff-bd47-c3def11ef235","Type":"ContainerStarted","Data":"8ceff4daf793551592086363ce30dac40963f69c85ec91ad67dc5505b664979d"}
Oct 30 21:13:32.062251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.062210    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" event={"ID":"98e23218-42fc-48c9-8b42-b3e56ce080f9","Type":"ContainerStarted","Data":"e25ebd1542cb8f44c87f25343e7659f36f8c1c7649a8cc6d0311a9b51bb2c641"}
Oct 30 21:13:32.062428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.062276    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" event={"ID":"98e23218-42fc-48c9-8b42-b3e56ce080f9","Type":"ContainerStarted","Data":"57589285c0d92a7ef6dc1af4d2fc1fb9891fb56390f248d0ff0c618914d62c5b"}
Oct 30 21:13:32.076416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.076384    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:13:32.105079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.104971    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podStartSLOduration=1.104952023 podStartE2EDuration="1.104952023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:32.101067392 +0000 UTC m=+755.512606829" watchObservedRunningTime="2026-10-30 21:13:32.104952023 +0000 UTC m=+755.516491460"
Oct 30 21:13:32.149513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.148665    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podStartSLOduration=2.148644954 podStartE2EDuration="2.148644954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:32.146854326 +0000 UTC m=+755.558393763" watchObservedRunningTime="2026-10-30 21:13:32.148644954 +0000 UTC m=+755.560184392"
Oct 30 21:13:32.193210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.193176    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:32.195160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.195126    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:32.195361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.195183    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:32.548813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.548708    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:32.550608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.550577    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:32.550727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:32.550640    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:33.193672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:33.193638    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:33.194071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:33.193700    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:33.550062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:33.549955    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:33.550342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:33.550314    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:33.713117 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:13:33.713080    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:13:34.193298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:34.193263    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:34.193476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:34.193324    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:34.548054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:34.547948    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:34.548682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:34.548650    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:35.193020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:35.192979    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:35.193199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:35.193043    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:35.551987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:35.551883    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:35.552634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:35.552591    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:36.193746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:36.193701    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:36.193939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:36.193779    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:36.554463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:36.554368    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:36.555084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:36.555052    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:37.193383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:37.193352    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:37.193617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:37.193399    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 21:13:37.548051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:37.547951    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:37.548332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:37.548291    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:38.547038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:38.547006    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:13:38.547541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:38.547511    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:13:39.234707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.234665    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:13:39.235022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.234990    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:39.654704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.654654    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:39.656719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.656120    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:39.660150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.660124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:13:39.813840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.813802    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:39.815642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.815442    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:39.817505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:39.817469    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:13:58.370449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.370404    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:13:58.376325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.376301    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.390705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.390657    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.390953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.390923    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhrw\" (UniqueName: \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.391086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.391014    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.391155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.391115    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.391212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.391177    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.391290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.391273    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.394245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.394214    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:13:58.401624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.401583    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-58fea9\"/\"distinct-2-w0-58fea9-kserve-self-signed-certs\""
Oct 30 21:13:58.492237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492201    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492239    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhrw\" (UniqueName: \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492294    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492369    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492423    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492531    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492743    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492800    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.492914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.492889    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.494910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.494880    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.495042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.495024    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.511780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.511748    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhrw\" (UniqueName: \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\") pod \"distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") " pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:58.696641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:58.696536    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:13:59.009153 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:13:59.009115    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1d52d9_d0cd_4322_969b_3a3e841db30f.slice/crio-7a71b1f0648331989182fe2efde1d26732f8c5a3f77dbb0575b4bdd6055726d1 WatchSource:0}: Error finding container 7a71b1f0648331989182fe2efde1d26732f8c5a3f77dbb0575b4bdd6055726d1: Status 404 returned error can't find the container with id 7a71b1f0648331989182fe2efde1d26732f8c5a3f77dbb0575b4bdd6055726d1
Oct 30 21:13:59.010724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:59.010697    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:13:59.295545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:59.295425    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" event={"ID":"6e1d52d9-d0cd-4322-969b-3a3e841db30f","Type":"ContainerStarted","Data":"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"}
Oct 30 21:13:59.295545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:59.295484    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" event={"ID":"6e1d52d9-d0cd-4322-969b-3a3e841db30f","Type":"ContainerStarted","Data":"7a71b1f0648331989182fe2efde1d26732f8c5a3f77dbb0575b4bdd6055726d1"}
Oct 30 21:13:59.339515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:59.339369    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podStartSLOduration=1.339347241 podStartE2EDuration="1.339347241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:59.338243693 +0000 UTC m=+782.749783130" watchObservedRunningTime="2026-10-30 21:13:59.339347241 +0000 UTC m=+782.750886680"
Oct 30 21:13:59.698511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:13:59.697667    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:14:03.672811 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:14:03.672769    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:14:07.996253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:07.996217    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"]
Oct 30 21:14:08.001697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.001674    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.018456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.018429    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"]
Oct 30 21:14:08.026380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.026342    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:08.026647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.026624    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:14:08.083248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.083210    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.083434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.083281    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.083434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.083353    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgv6\" (UniqueName: \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.184615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.184575    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.184802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.184645    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.184802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.184667    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgv6\" (UniqueName: \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.185437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.185407    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.187054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.187031    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.208568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.208527    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgv6\" (UniqueName: \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\") pod \"maas-api-revoke-sub-71d088bc-2gl9r\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.271574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.271457    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:14:08.275950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.275922    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.297336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.293587    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2cda44\"/\"distinct-2-w2-2cda44-kserve-self-signed-certs\""
Oct 30 21:14:08.297336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.297109    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:14:08.316809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.316768    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:08.387674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387636    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.387815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387716    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.387815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387745    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mpk\" (UniqueName: \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.387944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387869    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.387944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387900    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.387944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.387928    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491183    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491261    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491288    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mpk\" (UniqueName: \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491415    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491446    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491472    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.491884    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.492172    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.492571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.492382    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.515134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.512289    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.519981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.519937    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.523200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.523107    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mpk\" (UniqueName: \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\") pod \"distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") " pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.593663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.593614    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:08.633090 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:08.633046    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018a1388_b8f3_4ad0_945d_4ec326d4b56c.slice/crio-9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858 WatchSource:0}: Error finding container 9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858: Status 404 returned error can't find the container with id 9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858
Oct 30 21:14:08.634057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.634031    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"]
Oct 30 21:14:08.697741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.697703    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:14:08.712776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.712743    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:14:08.903291 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:08.903252    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod512c8a29_d681_4ef8_bc5c_8c21589dad70.slice/crio-53652bdf6e08d894ca6f975bbbd2a4628e329c40189f7aa857b6591f210c4541 WatchSource:0}: Error finding container 53652bdf6e08d894ca6f975bbbd2a4628e329c40189f7aa857b6591f210c4541: Status 404 returned error can't find the container with id 53652bdf6e08d894ca6f975bbbd2a4628e329c40189f7aa857b6591f210c4541
Oct 30 21:14:08.904154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:08.904065    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:14:09.461413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.461361    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r" event={"ID":"018a1388-b8f3-4ad0-945d-4ec326d4b56c","Type":"ContainerStarted","Data":"9b943fd2dc0ce3dddd961377e565e4c7b4466632bfa1b43019964f080f271f88"}
Oct 30 21:14:09.461939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.461479    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r" event={"ID":"018a1388-b8f3-4ad0-945d-4ec326d4b56c","Type":"ContainerStarted","Data":"9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858"}
Oct 30 21:14:09.463785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.463750    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" event={"ID":"512c8a29-d681-4ef8-bc5c-8c21589dad70","Type":"ContainerStarted","Data":"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"}
Oct 30 21:14:09.463899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.463793    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" event={"ID":"512c8a29-d681-4ef8-bc5c-8c21589dad70","Type":"ContainerStarted","Data":"53652bdf6e08d894ca6f975bbbd2a4628e329c40189f7aa857b6591f210c4541"}
Oct 30 21:14:09.475088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.475052    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:14:09.502673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.502613    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r" podStartSLOduration=2.502598895 podStartE2EDuration="2.502598895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:09.500127193 +0000 UTC m=+792.911666630" watchObservedRunningTime="2026-10-30 21:14:09.502598895 +0000 UTC m=+792.914138331"
Oct 30 21:14:09.581808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.581749    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podStartSLOduration=1.5817332240000002 podStartE2EDuration="1.581733224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:09.578671087 +0000 UTC m=+792.990210526" watchObservedRunningTime="2026-10-30 21:14:09.581733224 +0000 UTC m=+792.993272661"
Oct 30 21:14:09.594671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:09.594636    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:10.472803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:10.472767    7037 generic.go:356] "Generic (PLEG): container finished" podID="018a1388-b8f3-4ad0-945d-4ec326d4b56c" containerID="9b943fd2dc0ce3dddd961377e565e4c7b4466632bfa1b43019964f080f271f88" exitCode=0
Oct 30 21:14:10.473305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:10.472853    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r" event={"ID":"018a1388-b8f3-4ad0-945d-4ec326d4b56c","Type":"ContainerDied","Data":"9b943fd2dc0ce3dddd961377e565e4c7b4466632bfa1b43019964f080f271f88"}
Oct 30 21:14:11.634390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.634362    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:11.738003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.737914    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\" (UniqueName: \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\") pod \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") "
Oct 30 21:14:11.738003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.737976    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\") pod \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") "
Oct 30 21:14:11.738003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.737996    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\") pod \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\" (UID: \"018a1388-b8f3-4ad0-945d-4ec326d4b56c\") "
Oct 30 21:14:11.738411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.738372    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca" pod "018a1388-b8f3-4ad0-945d-4ec326d4b56c" (UID: "018a1388-b8f3-4ad0-945d-4ec326d4b56c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:11.738536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.738506    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:11.740295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.740266    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca" pod "018a1388-b8f3-4ad0-945d-4ec326d4b56c" (UID: "018a1388-b8f3-4ad0-945d-4ec326d4b56c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:11.740295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.740270    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6" pod "018a1388-b8f3-4ad0-945d-4ec326d4b56c" (UID: "018a1388-b8f3-4ad0-945d-4ec326d4b56c"). InnerVolumeSpecName "kube-api-access-ztgv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:11.839570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.839539    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgv6\" (UniqueName: \"kubernetes.io/projected/018a1388-b8f3-4ad0-945d-4ec326d4b56c-kube-api-access-ztgv6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:11.839570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:11.839568    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/018a1388-b8f3-4ad0-945d-4ec326d4b56c-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:12.489779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:12.489747    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r" event={"ID":"018a1388-b8f3-4ad0-945d-4ec326d4b56c","Type":"ContainerDied","Data":"9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858"}
Oct 30 21:14:12.489956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:12.489781    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"
Oct 30 21:14:12.489956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:12.489793    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e60e75afd16f5f0e740324c159f2062fc35e3f176155665f1b11f38dbc3d858"
Oct 30 21:14:15.579143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.579105    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:14:15.579989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.579965    7037 cpu_manager.go:414] "containerMap: removing container" podUID="018a1388-b8f3-4ad0-945d-4ec326d4b56c" containerName="revoke-keys"
Oct 30 21:14:15.580094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.579990    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018a1388-b8f3-4ad0-945d-4ec326d4b56c" containerName="revoke-keys"
Oct 30 21:14:15.580156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.580147    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="018a1388-b8f3-4ad0-945d-4ec326d4b56c" containerName="revoke-keys"
Oct 30 21:14:15.583864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.583842    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.593338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.593305    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:14:15.598699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.598677    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.615956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.615919    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:14:15.618449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.618414    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.618723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.618698    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.619554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619466    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619551    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619588    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619623    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619673    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619711    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619787    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619812    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619860    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619877    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmg7\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619901    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619923    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619945    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.619982    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.620048    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.620121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.620076    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv96f\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.652832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.652797    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:14:15.657893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.657846    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:14:15.666560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.663427    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.666560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.666279    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:14:15.721936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.721899    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722020    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722054    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722079    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722100    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722285    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722339    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmg7\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722409    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722446    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv96f\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722474    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722613    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722669    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722707    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722753    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722888    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.722970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722918    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.722970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722939    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.722970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722967    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.722985    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723020    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq59j\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.723126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723041    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723069    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723167    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723176    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723208    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723219    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723241    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723280    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723358    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723394    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.723661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723480    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723527    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.723882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723752    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.723882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.723854    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.724350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.724327    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.724880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.724859    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.725250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.725231    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.725786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.725769    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.726133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.726110    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.742542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.742508    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmg7\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.742691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.742612    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.742691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.742648    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.742825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.742781    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv96f\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-m595f\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.825766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.825719    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.825924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.825817    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.825924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.825863    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq59j\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.825924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.825903    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826015    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826068    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826142    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826153    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826173    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826291    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826300    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826380    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826543    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.826874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.826854    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.828619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.828575    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.828739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.828700    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.846565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.846467    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.846706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.846687    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq59j\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:15.900995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.900958    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:15.920480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.920448    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:15.983580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:15.983257    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:16.230092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.229992    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:14:16.238030 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:16.238001    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9156b2d8_9d48_4320_817e_29fa49050c5d.slice/crio-fddf1962034cf3b9ff73ccf6a43984f1cdd95395d757aeb73c96a8ffc8ebd07a WatchSource:0}: Error finding container fddf1962034cf3b9ff73ccf6a43984f1cdd95395d757aeb73c96a8ffc8ebd07a: Status 404 returned error can't find the container with id fddf1962034cf3b9ff73ccf6a43984f1cdd95395d757aeb73c96a8ffc8ebd07a
Oct 30 21:14:16.240505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.240456    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.240607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.240555    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.240674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.240605    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.263807 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:16.263778    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59227f7_56e1_4111_982c_2d1c3713368b.slice/crio-fb8c4b5eff7b91b69cfd68035b13a1c8eb68ced6aeb3a7c9ec946f4d9430600b WatchSource:0}: Error finding container fb8c4b5eff7b91b69cfd68035b13a1c8eb68ced6aeb3a7c9ec946f4d9430600b: Status 404 returned error can't find the container with id fb8c4b5eff7b91b69cfd68035b13a1c8eb68ced6aeb3a7c9ec946f4d9430600b
Oct 30 21:14:16.266550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.266478    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:14:16.266656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.266501    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.266656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.266625    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.266726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.266662    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.304828 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:16.304790    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1e9aec_c264_4d18_a5af_f930f8fdea14.slice/crio-8c8b1771c42d592bb050bc617c6a7a3df27770af15292143b3dc27601447a2e5 WatchSource:0}: Error finding container 8c8b1771c42d592bb050bc617c6a7a3df27770af15292143b3dc27601447a2e5: Status 404 returned error can't find the container with id 8c8b1771c42d592bb050bc617c6a7a3df27770af15292143b3dc27601447a2e5
Oct 30 21:14:16.305864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.305842    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:14:16.308349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.308318    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.308403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.308388    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.308440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.308418    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:16.540832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.540730    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" event={"ID":"9156b2d8-9d48-4320-817e-29fa49050c5d","Type":"ContainerStarted","Data":"361c6defbcc6deec49bb18dcc7b8d752e530a8dd7136ef9769b45cc7ec887ee0"}
Oct 30 21:14:16.540992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.540838    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" event={"ID":"9156b2d8-9d48-4320-817e-29fa49050c5d","Type":"ContainerStarted","Data":"fddf1962034cf3b9ff73ccf6a43984f1cdd95395d757aeb73c96a8ffc8ebd07a"}
Oct 30 21:14:16.544131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.544098    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" event={"ID":"b59227f7-56e1-4111-982c-2d1c3713368b","Type":"ContainerStarted","Data":"bab208a68c088b8fb7f6122302fd2b9245fee6e3f9f3c174359c303eb04e223f"}
Oct 30 21:14:16.544252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.544156    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" event={"ID":"b59227f7-56e1-4111-982c-2d1c3713368b","Type":"ContainerStarted","Data":"fb8c4b5eff7b91b69cfd68035b13a1c8eb68ced6aeb3a7c9ec946f4d9430600b"}
Oct 30 21:14:16.546771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.546745    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" event={"ID":"de1e9aec-c264-4d18-a5af-f930f8fdea14","Type":"ContainerStarted","Data":"d343a82a8d9d19ea8ae939e77b2ca8d722ad573d647401a0af418002a6a3f96f"}
Oct 30 21:14:16.546879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.546792    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" event={"ID":"de1e9aec-c264-4d18-a5af-f930f8fdea14","Type":"ContainerStarted","Data":"8c8b1771c42d592bb050bc617c6a7a3df27770af15292143b3dc27601447a2e5"}
Oct 30 21:14:16.727438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.727370    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podStartSLOduration=1.7273504229999999 podStartE2EDuration="1.727350423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.725407728 +0000 UTC m=+800.136947166" watchObservedRunningTime="2026-10-30 21:14:16.727350423 +0000 UTC m=+800.138889861"
Oct 30 21:14:16.730563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.730517    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podStartSLOduration=1.7304852990000001 podStartE2EDuration="1.730485299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.645954136 +0000 UTC m=+800.057493569" watchObservedRunningTime="2026-10-30 21:14:16.730485299 +0000 UTC m=+800.142024737"
Oct 30 21:14:16.826129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.826061    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podStartSLOduration=1.826040936 podStartE2EDuration="1.826040936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.819064065 +0000 UTC m=+800.230603498" watchObservedRunningTime="2026-10-30 21:14:16.826040936 +0000 UTC m=+800.237580374"
Oct 30 21:14:16.901774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.901723    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:16.921349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.921314    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:16.992421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:16.992389    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:17.987515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:17.987003    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:17.987515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:17.987062    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:17.987515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:17.987364    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:17.987515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:17.987399    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:18.008041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.008002    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:18.008260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.008062    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:18.601237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.598956    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:18.614873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.614845    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:18.717039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.717008    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:14:18.913709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.913597    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:18.915057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.915000    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:18.929522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.926648    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:18.929522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:18.926730    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:19.070533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.061445    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.070533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.061542    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:19.904879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.904837    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.905057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.904901    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:19.921330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.921266    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.921330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.921323    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:19.985668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.985612    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.985856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:19.985692    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:20.904229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.903609    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:20.904229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.903672    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:20.920835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.920799    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:20.921007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.920857    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:20.989516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.989237    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:20.989516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:20.989291    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:21.904024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.903974    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:21.904256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.904038    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:21.921540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.921476    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:21.921696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.921552    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:21.993517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.984922    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:21.993517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:21.984985    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:22.901657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.901603    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:14:22.901901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.901680    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 21:14:22.921681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.921641    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:14:22.922208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.921720    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:14:22.984124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.984086    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:14:22.984334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:22.984151    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 21:14:23.907480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:23.907444    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:23.907926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:23.907889    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:23.908847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:23.908824    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:14:24.252269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.252170    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:24.253680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.253625    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:24.254700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.254681    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:14:24.480909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.480880    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:24.481269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.481245    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:24.482099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:24.482080    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:14:31.628054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.628019    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"]
Oct 30 21:14:31.634209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.634173    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.666951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.666915    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:31.667129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.666985    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:14:31.673805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.673763    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"]
Oct 30 21:14:31.707390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.707360    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.707555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.707516    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.707705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.707678    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7jf\" (UniqueName: \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.808669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.808634    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7jf\" (UniqueName: \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.808881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.808746    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.808958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.808900    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.809556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.809517    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.811292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.811267    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.834690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.834663    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7jf\" (UniqueName: \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\") pod \"maas-api-revoke-sub-fbd05764-qlb2k\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:31.983070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:31.982980    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:32.281206 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:32.281168    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7909e148_60bf_4f69_9dc9_857230751c39.slice/crio-c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5 WatchSource:0}: Error finding container c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5: Status 404 returned error can't find the container with id c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5
Oct 30 21:14:32.283417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:32.283392    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"]
Oct 30 21:14:32.810578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:32.810446    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k" event={"ID":"7909e148-60bf-4f69-9dc9-857230751c39","Type":"ContainerStarted","Data":"55a56b09f5bf41e1ba62200dd8a6646328231e0fc5f354f257cf037492232a80"}
Oct 30 21:14:32.810578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:32.810519    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k" event={"ID":"7909e148-60bf-4f69-9dc9-857230751c39","Type":"ContainerStarted","Data":"c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5"}
Oct 30 21:14:32.852966 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:32.852920    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k" podStartSLOduration=1.852908635 podStartE2EDuration="1.852908635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:32.852305794 +0000 UTC m=+816.263845230" watchObservedRunningTime="2026-10-30 21:14:32.852908635 +0000 UTC m=+816.264448071"
Oct 30 21:14:33.672799 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:14:33.672761    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:14:33.821452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:33.821415    7037 generic.go:356] "Generic (PLEG): container finished" podID="7909e148-60bf-4f69-9dc9-857230751c39" containerID="55a56b09f5bf41e1ba62200dd8a6646328231e0fc5f354f257cf037492232a80" exitCode=0
Oct 30 21:14:33.821452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:33.821443    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k" event={"ID":"7909e148-60bf-4f69-9dc9-857230751c39","Type":"ContainerDied","Data":"55a56b09f5bf41e1ba62200dd8a6646328231e0fc5f354f257cf037492232a80"}
Oct 30 21:14:34.992261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:34.992237    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:35.055768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.055716    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\" (UniqueName: \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\") pod \"7909e148-60bf-4f69-9dc9-857230751c39\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") "
Oct 30 21:14:35.055962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.055873    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\") pod \"7909e148-60bf-4f69-9dc9-857230751c39\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") "
Oct 30 21:14:35.055962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.055937    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\") pod \"7909e148-60bf-4f69-9dc9-857230751c39\" (UID: \"7909e148-60bf-4f69-9dc9-857230751c39\") "
Oct 30 21:14:35.056587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.056556    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca" pod "7909e148-60bf-4f69-9dc9-857230751c39" (UID: "7909e148-60bf-4f69-9dc9-857230751c39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:35.056756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.056740    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:35.057950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.057919    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf" pod "7909e148-60bf-4f69-9dc9-857230751c39" (UID: "7909e148-60bf-4f69-9dc9-857230751c39"). InnerVolumeSpecName "kube-api-access-jx7jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:35.058540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.058516    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca" pod "7909e148-60bf-4f69-9dc9-857230751c39" (UID: "7909e148-60bf-4f69-9dc9-857230751c39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:35.158551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.158434    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7jf\" (UniqueName: \"kubernetes.io/projected/7909e148-60bf-4f69-9dc9-857230751c39-kube-api-access-jx7jf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:35.158551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.158469    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7909e148-60bf-4f69-9dc9-857230751c39-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:35.839011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.838932    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k" event={"ID":"7909e148-60bf-4f69-9dc9-857230751c39","Type":"ContainerDied","Data":"c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5"}
Oct 30 21:14:35.839011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.838984    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b9c4ce6508ee6f7822b72310730fdfc199c7a8cb521954bf33ba4a6dc175f5"
Oct 30 21:14:35.839011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:35.838941    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"
Oct 30 21:14:41.249333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.249288    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"]
Oct 30 21:14:41.251019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.250989    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7909e148-60bf-4f69-9dc9-857230751c39" containerName="revoke-keys"
Oct 30 21:14:41.251188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.251173    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7909e148-60bf-4f69-9dc9-857230751c39" containerName="revoke-keys"
Oct 30 21:14:41.251510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.251475    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7909e148-60bf-4f69-9dc9-857230751c39" containerName="revoke-keys"
Oct 30 21:14:41.300183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.300150    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"]
Oct 30 21:14:41.300635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.300542    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.321580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.320807    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:14:41.483516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.433864    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:41.483516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.458230    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.483516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.458484    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zg6\" (UniqueName: \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.483516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.458543    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.623409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.604221    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zg6\" (UniqueName: \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.623409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.604292    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.623409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.604459    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.623409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.605795    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.729515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.696913    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.769516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.730460    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zg6\" (UniqueName: \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\") pod \"maas-api-revoke-sub-3ae50bfe-pc6np\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:41.870845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:41.870813    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:42.126727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.126682    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.292810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.292708    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.542555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.484412    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.542555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.489098    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:14:42.591530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.587399    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.593654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.593609    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.654381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.636816    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.889988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.889875    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:42.948615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:42.948473    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:14:43.068336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.067942    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:43.173527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.111644    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:14:43.173971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.160283    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:43.198631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.172577    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:43.306971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.306929    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:43.324088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.322785    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:14:43.324088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.322866    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:14:43.349057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.348619    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"]
Oct 30 21:14:43.440569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.429485    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:14:43.643483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:43.641254    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:14:44.228824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:44.228721    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np" event={"ID":"551bc85c-a40c-4a30-9227-c28f0bde1819","Type":"ContainerStarted","Data":"741a90b9a0b3776b33be85cebc4377254fc2668e6a538aab0575595d92e0e75c"}
Oct 30 21:14:44.228824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:44.228793    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np" event={"ID":"551bc85c-a40c-4a30-9227-c28f0bde1819","Type":"ContainerStarted","Data":"f6d4cb16f5579ba2f8952bc51ff523b741d2a8d971e02c5037078cd5aac15457"}
Oct 30 21:14:44.271785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:44.271725    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np" podStartSLOduration=3.271709299 podStartE2EDuration="3.271709299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:44.267534579 +0000 UTC m=+827.679074016" watchObservedRunningTime="2026-10-30 21:14:44.271709299 +0000 UTC m=+827.683248736"
Oct 30 21:14:45.236935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.236898    7037 generic.go:356] "Generic (PLEG): container finished" podID="551bc85c-a40c-4a30-9227-c28f0bde1819" containerID="741a90b9a0b3776b33be85cebc4377254fc2668e6a538aab0575595d92e0e75c" exitCode=0
Oct 30 21:14:45.237346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.236982    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np" event={"ID":"551bc85c-a40c-4a30-9227-c28f0bde1819","Type":"ContainerDied","Data":"741a90b9a0b3776b33be85cebc4377254fc2668e6a538aab0575595d92e0e75c"}
Oct 30 21:14:45.575391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.575357    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:14:45.579835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.579813    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.605868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.605833    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:14:45.623712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.623654    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:14:45.632933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.632905    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.664676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.664643    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:14:45.669386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669351    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669418    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.669599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669452    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669524    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn26\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.669599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669557    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwftc\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669583    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669624    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.669898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669670    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.669898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669733    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669780    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.669898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669814    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.670139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669925    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.670139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669951    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.670139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.669994    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.670139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.670050    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.670139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.670109    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.670405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.670156    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.670405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.670201    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.771806    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.771876    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.771911    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772014    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772047    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772106    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772194    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772297    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772352    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772396    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772430    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772511    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772547    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772616    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn26\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772643    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwftc\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772671    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772719    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.772774    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.773807    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.774124    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.774606    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.775000    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.775015    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.775270    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.776002    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.776539    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.776795    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.778612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.777375    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.782550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.782510    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.783740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.782807    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.783740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.783681    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.785183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.785165    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.793286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.793259    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.793780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.793759    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.794326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.794308    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn26\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.794764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.794747    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwftc\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:45.894716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.894621    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:45.962468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:45.962442    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:46.220871 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:46.220838    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf432a47_d0e6_474d_91b4_78c1b5518385.slice/crio-3c49d4f48da33ccdbb4439f3a8c7258d6f93b46e325c5d8d9600426ec58d382a WatchSource:0}: Error finding container 3c49d4f48da33ccdbb4439f3a8c7258d6f93b46e325c5d8d9600426ec58d382a: Status 404 returned error can't find the container with id 3c49d4f48da33ccdbb4439f3a8c7258d6f93b46e325c5d8d9600426ec58d382a
Oct 30 21:14:46.222782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.222755    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:14:46.223632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.223592    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.223738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.223676    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.223738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.223724    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.246033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.246006    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" event={"ID":"df432a47-d0e6-474d-91b4-78c1b5518385","Type":"ContainerStarted","Data":"3c49d4f48da33ccdbb4439f3a8c7258d6f93b46e325c5d8d9600426ec58d382a"}
Oct 30 21:14:46.299637 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:46.299598    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ac3d5_ebdf_45b5_aab5_5f14be8e2aa2.slice/crio-3ac3abe4796b5c8df79c61649d4e29221f1d58f1dd861c933c73f36a673b6154 WatchSource:0}: Error finding container 3ac3abe4796b5c8df79c61649d4e29221f1d58f1dd861c933c73f36a673b6154: Status 404 returned error can't find the container with id 3ac3abe4796b5c8df79c61649d4e29221f1d58f1dd861c933c73f36a673b6154
Oct 30 21:14:46.302343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.302314    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:14:46.302827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.302706    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.302827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.302792    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.302992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.302843    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:14:46.471944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.471861    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:46.487787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.486362    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\") pod \"551bc85c-a40c-4a30-9227-c28f0bde1819\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") "
Oct 30 21:14:46.487787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.486557    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\" (UniqueName: \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\") pod \"551bc85c-a40c-4a30-9227-c28f0bde1819\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") "
Oct 30 21:14:46.487787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.486744    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\") pod \"551bc85c-a40c-4a30-9227-c28f0bde1819\" (UID: \"551bc85c-a40c-4a30-9227-c28f0bde1819\") "
Oct 30 21:14:46.489306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.489267    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca" pod "551bc85c-a40c-4a30-9227-c28f0bde1819" (UID: "551bc85c-a40c-4a30-9227-c28f0bde1819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:46.491134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.491105    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca" pod "551bc85c-a40c-4a30-9227-c28f0bde1819" (UID: "551bc85c-a40c-4a30-9227-c28f0bde1819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:46.492671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.492630    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6" pod "551bc85c-a40c-4a30-9227-c28f0bde1819" (UID: "551bc85c-a40c-4a30-9227-c28f0bde1819"). InnerVolumeSpecName "kube-api-access-27zg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:46.588584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.588547    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zg6\" (UniqueName: \"kubernetes.io/projected/551bc85c-a40c-4a30-9227-c28f0bde1819-kube-api-access-27zg6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:46.588584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.588578    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:46.588584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:46.588588    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551bc85c-a40c-4a30-9227-c28f0bde1819-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:47.284353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.284325    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"
Oct 30 21:14:47.284978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.284324    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np" event={"ID":"551bc85c-a40c-4a30-9227-c28f0bde1819","Type":"ContainerDied","Data":"f6d4cb16f5579ba2f8952bc51ff523b741d2a8d971e02c5037078cd5aac15457"}
Oct 30 21:14:47.285067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.285046    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6d4cb16f5579ba2f8952bc51ff523b741d2a8d971e02c5037078cd5aac15457"
Oct 30 21:14:47.287007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.286960    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" event={"ID":"df432a47-d0e6-474d-91b4-78c1b5518385","Type":"ContainerStarted","Data":"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"}
Oct 30 21:14:47.290298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.290222    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" event={"ID":"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2","Type":"ContainerStarted","Data":"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"}
Oct 30 21:14:47.290298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.290267    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" event={"ID":"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2","Type":"ContainerStarted","Data":"3ac3abe4796b5c8df79c61649d4e29221f1d58f1dd861c933c73f36a673b6154"}
Oct 30 21:14:47.383727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.378585    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podStartSLOduration=2.378566547 podStartE2EDuration="2.378566547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:47.35007279 +0000 UTC m=+830.761612240" watchObservedRunningTime="2026-10-30 21:14:47.378566547 +0000 UTC m=+830.790105985"
Oct 30 21:14:47.432914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.432848    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podStartSLOduration=2.432824301 podStartE2EDuration="2.432824301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:47.422034374 +0000 UTC m=+830.833573812" watchObservedRunningTime="2026-10-30 21:14:47.432824301 +0000 UTC m=+830.844363739"
Oct 30 21:14:47.895562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.895529    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:47.897419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.897393    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:47.897569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.897448    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:47.968521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.967886    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:47.972513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.969631    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:47.972513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:47.969725    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:48.896021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:48.895982    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:48.896556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:48.896472    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:48.972603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:48.970980    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:48.972603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:48.971044    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:49.895217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:49.895173    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:49.895400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:49.895241    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:49.973516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:49.970378    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:49.979681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:49.975559    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:50.895008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:50.894969    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:50.895178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:50.895101    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:50.984714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:50.984675    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:50.987668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:50.987621    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:51.895255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:51.895219    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:51.895462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:51.895276    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:51.969369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:51.969327    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:51.969688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:51.969655    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:52.895991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:52.895953    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:14:52.896399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:52.896019    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 21:14:52.963217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:52.963177    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:14:52.963380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:52.963257    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 21:14:54.013734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.013690    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:54.014894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.014831    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:54.016537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.016484    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:14:54.157937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.157800    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:54.158607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.158295    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:54.160094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:54.160070    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:14:55.512675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.512644    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"]
Oct 30 21:14:55.513389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.513369    7037 cpu_manager.go:414] "containerMap: removing container" podUID="551bc85c-a40c-4a30-9227-c28f0bde1819" containerName="revoke-keys"
Oct 30 21:14:55.513450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.513394    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bc85c-a40c-4a30-9227-c28f0bde1819" containerName="revoke-keys"
Oct 30 21:14:55.513604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.513590    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bc85c-a40c-4a30-9227-c28f0bde1819" containerName="revoke-keys"
Oct 30 21:14:55.518528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.518506    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.524810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.524789    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgcl\" (UniqueName: \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.524913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.524856    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.525012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.524990    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.541430 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.541411    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:55.541705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.541692    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:14:55.558843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.558817    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"]
Oct 30 21:14:55.625924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.625891    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgcl\" (UniqueName: \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.626120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.625961    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.626120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.626039    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.626720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.626694    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.628396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.628369    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.667168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.667135    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgcl\" (UniqueName: \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\") pod \"maas-api-revoke-sub-602075ad-rlsnz\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:55.833336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:55.833308    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:56.132806 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:14:56.132769    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65e59b5_1012_49ac_a0f8_5ec66c09c9ac.slice/crio-291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0 WatchSource:0}: Error finding container 291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0: Status 404 returned error can't find the container with id 291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0
Oct 30 21:14:56.136113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:56.136090    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"]
Oct 30 21:14:56.396281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:56.396193    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz" event={"ID":"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac","Type":"ContainerStarted","Data":"291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0"}
Oct 30 21:14:57.404873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:57.404840    7037 generic.go:356] "Generic (PLEG): container finished" podID="f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" containerID="88bce672de8f766ab2868ec7e36d4643da24b9732bd42d0f3b7b26b2a4618632" exitCode=0
Oct 30 21:14:57.405306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:57.404905    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz" event={"ID":"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac","Type":"ContainerDied","Data":"88bce672de8f766ab2868ec7e36d4643da24b9732bd42d0f3b7b26b2a4618632"}
Oct 30 21:14:58.563820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.563794    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:58.668162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.668079    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\") pod \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") "
Oct 30 21:14:58.668162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.668139    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\") pod \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") "
Oct 30 21:14:58.668350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.668216    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\" (UniqueName: \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\") pod \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\" (UID: \"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac\") "
Oct 30 21:14:58.668502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.668465    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca" pod "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" (UID: "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:58.668800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.668774    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:58.670183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.670159    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca" pod "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" (UID: "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:58.670276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.670262    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl" pod "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" (UID: "f65e59b5-1012-49ac-a0f8-5ec66c09c9ac"). InnerVolumeSpecName "kube-api-access-2vgcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:58.776085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.771228    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgcl\" (UniqueName: \"kubernetes.io/projected/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-kube-api-access-2vgcl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:58.776085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:58.771301    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:14:59.419848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:59.419820    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"
Oct 30 21:14:59.420038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:59.419856    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz" event={"ID":"f65e59b5-1012-49ac-a0f8-5ec66c09c9ac","Type":"ContainerDied","Data":"291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0"}
Oct 30 21:14:59.420038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:14:59.419925    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291c11f93f96cb732e2836c98c65ad3a829e98b8efa63a0372219cc78f4ca8f0"
Oct 30 21:15:00.154867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.154831    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:00.155806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.155780    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" containerName="revoke-keys"
Oct 30 21:15:00.155806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.155807    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" containerName="revoke-keys"
Oct 30 21:15:00.155971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.155959    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" containerName="revoke-keys"
Oct 30 21:15:00.159923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.159900    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.161296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.161271    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:00.166043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.166027    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.172814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.172794    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:00.177170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.177151    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.184564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.184540    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:15:00.190212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.190187    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.190303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.190253    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:00.190303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.190290    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcf2\" (UniqueName: \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.190387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.190358    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrr7\" (UniqueName: \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.190421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.190385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.196397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.196078    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:00.204605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.204583    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:00.204709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.204673    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.207545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.207525    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:00.231563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.231533    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:00.291820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.291786    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mq6\" (UniqueName: \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.292018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.291834    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.292091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292010    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9hk\" (UniqueName: \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.292132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292116    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.292236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292219    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.292300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292274    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcf2\" (UniqueName: \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.292357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292328    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrr7\" (UniqueName: \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.292480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292455    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.292775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.292748    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.293097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.293046    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.312597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.312574    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrr7\" (UniqueName: \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.312702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.312666    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcf2\" (UniqueName: \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.393569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.393536    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mq6\" (UniqueName: \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.393733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.393592    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.393733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.393702    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9hk\" (UniqueName: \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.393831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.393807    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.394325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.394300    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.394442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.394422    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.413144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.413075    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9hk\" (UniqueName: \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\") pod \"maas-api-key-cleanup-29889915-vzrnd\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.413144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.413077    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mq6\" (UniqueName: \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.477391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.477358    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:00.487176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.487150    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:00.499501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.499463    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:00.522925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.522894    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:00.806862 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:00.806829    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d0eeb7_5cf4_4f3d_af7e_f6918f7e1557.slice/crio-315ba57bab029e0f55c251e4a0efff03967229e950c53dcc1089b596ec1ce9c9 WatchSource:0}: Error finding container 315ba57bab029e0f55c251e4a0efff03967229e950c53dcc1089b596ec1ce9c9: Status 404 returned error can't find the container with id 315ba57bab029e0f55c251e4a0efff03967229e950c53dcc1089b596ec1ce9c9
Oct 30 21:15:00.809328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.809301    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:00.817304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.817277    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:00.820371 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:00.820341    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2cf21a_c93f_4eb7_8d3f_0168f8cd4d44.slice/crio-9a865f26bf80a0fd5b4815274e0b4d33c8227892197c7e7c60795d033910f4c3 WatchSource:0}: Error finding container 9a865f26bf80a0fd5b4815274e0b4d33c8227892197c7e7c60795d033910f4c3: Status 404 returned error can't find the container with id 9a865f26bf80a0fd5b4815274e0b4d33c8227892197c7e7c60795d033910f4c3
Oct 30 21:15:00.862870 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:00.862844    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95377ae4_1c7a_4681_a8a1_28b6062cb5ac.slice/crio-aa084f68c2198562f801715dc8ed01c5e151785da16e9051cefa1d95e3355fdc WatchSource:0}: Error finding container aa084f68c2198562f801715dc8ed01c5e151785da16e9051cefa1d95e3355fdc: Status 404 returned error can't find the container with id aa084f68c2198562f801715dc8ed01c5e151785da16e9051cefa1d95e3355fdc
Oct 30 21:15:00.865890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.865867    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:00.901924 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:00.901894    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef02bca8_e10c_4474_bda7_de776ef80bdb.slice/crio-f8c6da429d31723578e9079b6c320e289827b02940f92068c766201a27aaad0c WatchSource:0}: Error finding container f8c6da429d31723578e9079b6c320e289827b02940f92068c766201a27aaad0c: Status 404 returned error can't find the container with id f8c6da429d31723578e9079b6c320e289827b02940f92068c766201a27aaad0c
Oct 30 21:15:00.902176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:00.902152    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:01.444671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.444638    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerStarted","Data":"f8c6da429d31723578e9079b6c320e289827b02940f92068c766201a27aaad0c"}
Oct 30 21:15:01.446411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.446377    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerStarted","Data":"9a36be6755545ba03fe69c7b1325b5ba80dd05e561518ebc968eb7f7820d2453"}
Oct 30 21:15:01.446547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.446447    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerStarted","Data":"aa084f68c2198562f801715dc8ed01c5e151785da16e9051cefa1d95e3355fdc"}
Oct 30 21:15:01.447955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.447914    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerStarted","Data":"bae0a62baff990fb3be1c0dd869e299100e4db8f7a39d0f815e3708ceaf90810"}
Oct 30 21:15:01.448078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.447972    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerStarted","Data":"315ba57bab029e0f55c251e4a0efff03967229e950c53dcc1089b596ec1ce9c9"}
Oct 30 21:15:01.449333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.449308    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerStarted","Data":"3a29d12fa4e0a780504834262c24656f396e3a662e1174028415d1a7f09f886a"}
Oct 30 21:15:01.449421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.449347    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerStarted","Data":"9a865f26bf80a0fd5b4815274e0b4d33c8227892197c7e7c60795d033910f4c3"}
Oct 30 21:15:01.485278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.485179    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" podStartSLOduration=1.485166052 podStartE2EDuration="1.485166052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.483652364 +0000 UTC m=+844.895191800" watchObservedRunningTime="2026-10-30 21:15:01.485166052 +0000 UTC m=+844.896705523"
Oct 30 21:15:01.528479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.528417    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" podStartSLOduration=1.528392736 podStartE2EDuration="1.528392736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.522537475 +0000 UTC m=+844.934076913" watchObservedRunningTime="2026-10-30 21:15:01.528392736 +0000 UTC m=+844.939932176"
Oct 30 21:15:01.565290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:01.565232    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" podStartSLOduration=1.5652162600000001 podStartE2EDuration="1.56521626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.560832841 +0000 UTC m=+844.972372279" watchObservedRunningTime="2026-10-30 21:15:01.56521626 +0000 UTC m=+844.976755696"
Oct 30 21:15:02.461070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.461034    7037 generic.go:356] "Generic (PLEG): container finished" podID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerID="47d4bf9d92721fb1a8ac5b4976026faa319179d931c0b5fe7efe679ef7442941" exitCode=60
Oct 30 21:15:02.461517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.461119    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerDied","Data":"47d4bf9d92721fb1a8ac5b4976026faa319179d931c0b5fe7efe679ef7442941"}
Oct 30 21:15:02.461517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.461457    7037 scope.go:122] "RemoveContainer" containerID="47d4bf9d92721fb1a8ac5b4976026faa319179d931c0b5fe7efe679ef7442941"
Oct 30 21:15:02.462957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.462935    7037 generic.go:356] "Generic (PLEG): container finished" podID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerID="9a36be6755545ba03fe69c7b1325b5ba80dd05e561518ebc968eb7f7820d2453" exitCode=60
Oct 30 21:15:02.463060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.463009    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerDied","Data":"9a36be6755545ba03fe69c7b1325b5ba80dd05e561518ebc968eb7f7820d2453"}
Oct 30 21:15:02.463273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.463255    7037 scope.go:122] "RemoveContainer" containerID="9a36be6755545ba03fe69c7b1325b5ba80dd05e561518ebc968eb7f7820d2453"
Oct 30 21:15:02.469228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.469091    7037 generic.go:356] "Generic (PLEG): container finished" podID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerID="bae0a62baff990fb3be1c0dd869e299100e4db8f7a39d0f815e3708ceaf90810" exitCode=60
Oct 30 21:15:02.469291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.469236    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerDied","Data":"bae0a62baff990fb3be1c0dd869e299100e4db8f7a39d0f815e3708ceaf90810"}
Oct 30 21:15:02.469537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.469513    7037 scope.go:122] "RemoveContainer" containerID="bae0a62baff990fb3be1c0dd869e299100e4db8f7a39d0f815e3708ceaf90810"
Oct 30 21:15:02.471904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.471880    7037 generic.go:356] "Generic (PLEG): container finished" podID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerID="3a29d12fa4e0a780504834262c24656f396e3a662e1174028415d1a7f09f886a" exitCode=60
Oct 30 21:15:02.472024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.471963    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerDied","Data":"3a29d12fa4e0a780504834262c24656f396e3a662e1174028415d1a7f09f886a"}
Oct 30 21:15:02.472204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.472187    7037 scope.go:122] "RemoveContainer" containerID="3a29d12fa4e0a780504834262c24656f396e3a662e1174028415d1a7f09f886a"
Oct 30 21:15:02.534732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.534702    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"]
Oct 30 21:15:02.540443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.540420    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.560961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.560933    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:02.563764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.563672    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"]
Oct 30 21:15:02.628444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.628409    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrtt\" (UniqueName: \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.628763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.628746    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.629013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.628999    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.730059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.729984    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrtt\" (UniqueName: \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.730210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.730079    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.730262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.730226    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.730716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.730692    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.732502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.732470    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.750314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.750278    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrtt\" (UniqueName: \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\") pod \"maas-api-revoke-sub-c28bd106-rg6xz\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:02.865896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:02.865863    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:03.167348 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:03.167325    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c87b2a6_6e19_41eb_b1eb_95d28be054cd.slice/crio-982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4 WatchSource:0}: Error finding container 982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4: Status 404 returned error can't find the container with id 982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4
Oct 30 21:15:03.169521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.169482    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"]
Oct 30 21:15:03.483108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.483067    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerStarted","Data":"f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"}
Oct 30 21:15:03.485052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.485023    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerStarted","Data":"cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"}
Oct 30 21:15:03.486956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.486929    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerStarted","Data":"2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"}
Oct 30 21:15:03.488371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.488348    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz" event={"ID":"0c87b2a6-6e19-41eb-b1eb-95d28be054cd","Type":"ContainerStarted","Data":"982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4"}
Oct 30 21:15:03.490435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.490413    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerStarted","Data":"e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"}
Oct 30 21:15:03.524606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:03.524545    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" podStartSLOduration=3.524526507 podStartE2EDuration="3.524526507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.523006205 +0000 UTC m=+846.934545642" watchObservedRunningTime="2026-10-30 21:15:03.524526507 +0000 UTC m=+846.936065947"
Oct 30 21:15:03.673039 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:03.672999    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:15:04.500100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.500067    7037 generic.go:356] "Generic (PLEG): container finished" podID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerID="f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441" exitCode=60
Oct 30 21:15:04.500579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.500150    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerDied","Data":"f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"}
Oct 30 21:15:04.500579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.500221    7037 scope.go:122] "RemoveContainer" containerID="47d4bf9d92721fb1a8ac5b4976026faa319179d931c0b5fe7efe679ef7442941"
Oct 30 21:15:04.500579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.500532    7037 scope.go:122] "RemoveContainer" containerID="f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"
Oct 30 21:15:04.500979 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:04.500951    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-vzrnd_odh-ai-gateway-infra(ef02bca8-e10c-4474-bda7-de776ef80bdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb"
Oct 30 21:15:04.502422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.502400    7037 generic.go:356] "Generic (PLEG): container finished" podID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerID="cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957" exitCode=60
Oct 30 21:15:04.502574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.502484    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerDied","Data":"cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"}
Oct 30 21:15:04.502811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.502796    7037 scope.go:122] "RemoveContainer" containerID="cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"
Oct 30 21:15:04.503147 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:04.503123    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj_odh-ai-gateway-infra(95377ae4-1c7a-4681-a8a1-28b6062cb5ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac"
Oct 30 21:15:04.504752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.504724    7037 generic.go:356] "Generic (PLEG): container finished" podID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerID="2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887" exitCode=60
Oct 30 21:15:04.504829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.504799    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerDied","Data":"2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"}
Oct 30 21:15:04.505113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.505093    7037 scope.go:122] "RemoveContainer" containerID="2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"
Oct 30 21:15:04.505436 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:04.505410    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq_odh-ai-gateway-infra(87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557"
Oct 30 21:15:04.506651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.506628    7037 generic.go:356] "Generic (PLEG): container finished" podID="0c87b2a6-6e19-41eb-b1eb-95d28be054cd" containerID="af260e57a5101e6b86d9b5a69d281e54363f7d488776d8e4f665aa3b52534c43" exitCode=0
Oct 30 21:15:04.506756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.506656    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz" event={"ID":"0c87b2a6-6e19-41eb-b1eb-95d28be054cd","Type":"ContainerDied","Data":"af260e57a5101e6b86d9b5a69d281e54363f7d488776d8e4f665aa3b52534c43"}
Oct 30 21:15:04.509251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.509226    7037 generic.go:356] "Generic (PLEG): container finished" podID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerID="e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3" exitCode=60
Oct 30 21:15:04.509340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.509296    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerDied","Data":"e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"}
Oct 30 21:15:04.509584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.509567    7037 scope.go:122] "RemoveContainer" containerID="e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"
Oct 30 21:15:04.509870 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:04.509853    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64_odh-ai-gateway-infra(9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44"
Oct 30 21:15:04.515538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.515517    7037 scope.go:122] "RemoveContainer" containerID="9a36be6755545ba03fe69c7b1325b5ba80dd05e561518ebc968eb7f7820d2453"
Oct 30 21:15:04.530963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.530939    7037 scope.go:122] "RemoveContainer" containerID="bae0a62baff990fb3be1c0dd869e299100e4db8f7a39d0f815e3708ceaf90810"
Oct 30 21:15:04.545043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:04.544910    7037 scope.go:122] "RemoveContainer" containerID="3a29d12fa4e0a780504834262c24656f396e3a662e1174028415d1a7f09f886a"
Oct 30 21:15:05.521360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.521224    7037 scope.go:122] "RemoveContainer" containerID="e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"
Oct 30 21:15:05.521956 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:05.521855    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64_odh-ai-gateway-infra(9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44"
Oct 30 21:15:05.526544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.524361    7037 scope.go:122] "RemoveContainer" containerID="f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"
Oct 30 21:15:05.526544 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:05.524801    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-vzrnd_odh-ai-gateway-infra(ef02bca8-e10c-4474-bda7-de776ef80bdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb"
Oct 30 21:15:05.526793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.526566    7037 scope.go:122] "RemoveContainer" containerID="cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"
Oct 30 21:15:05.526923 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:05.526896    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj_odh-ai-gateway-infra(95377ae4-1c7a-4681-a8a1-28b6062cb5ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac"
Oct 30 21:15:05.530210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.530185    7037 scope.go:122] "RemoveContainer" containerID="2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"
Oct 30 21:15:05.530546 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:05.530521    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq_odh-ai-gateway-infra(87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557"
Oct 30 21:15:05.728598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.728559    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:05.785027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.784930    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\") pod \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") "
Oct 30 21:15:05.785207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.785098    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\") pod \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") "
Oct 30 21:15:05.785319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.785299    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\" (UniqueName: \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\") pod \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\" (UID: \"0c87b2a6-6e19-41eb-b1eb-95d28be054cd\") "
Oct 30 21:15:05.785386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.785312    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca" pod "0c87b2a6-6e19-41eb-b1eb-95d28be054cd" (UID: "0c87b2a6-6e19-41eb-b1eb-95d28be054cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:05.786303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.786273    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:05.787927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.787893    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca" pod "0c87b2a6-6e19-41eb-b1eb-95d28be054cd" (UID: "0c87b2a6-6e19-41eb-b1eb-95d28be054cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:05.788170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.788150    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt" pod "0c87b2a6-6e19-41eb-b1eb-95d28be054cd" (UID: "0c87b2a6-6e19-41eb-b1eb-95d28be054cd"). InnerVolumeSpecName "kube-api-access-wvrtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:05.887504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.887463    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrtt\" (UniqueName: \"kubernetes.io/projected/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-kube-api-access-wvrtt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:05.887504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:05.887510    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c87b2a6-6e19-41eb-b1eb-95d28be054cd-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:06.074051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.074012    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"]
Oct 30 21:15:06.075039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.075019    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0c87b2a6-6e19-41eb-b1eb-95d28be054cd" containerName="revoke-keys"
Oct 30 21:15:06.075133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.075040    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c87b2a6-6e19-41eb-b1eb-95d28be054cd" containerName="revoke-keys"
Oct 30 21:15:06.075251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.075240    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c87b2a6-6e19-41eb-b1eb-95d28be054cd" containerName="revoke-keys"
Oct 30 21:15:06.079577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.079553    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.093500 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.093457    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"]
Oct 30 21:15:06.192265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.192221    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.192454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.192438    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwnd\" (UniqueName: \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.192670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.192652    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.294229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.294195    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.294399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.294305    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwnd\" (UniqueName: \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.294437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.294423    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.294958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.294934    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.296781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.296758    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.313324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.313295    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwnd\" (UniqueName: \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\") pod \"maas-api-revoke-sub-ca5a6726-tb2s6\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.396415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.396338    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:06.542656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.542613    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz" event={"ID":"0c87b2a6-6e19-41eb-b1eb-95d28be054cd","Type":"ContainerDied","Data":"982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4"}
Oct 30 21:15:06.542656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.542653    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"
Oct 30 21:15:06.543069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.542676    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982cc89132d7dcb1409301382549f5ae903c1821c30f5f6b6dd01a070bfaa1f4"
Oct 30 21:15:06.694600 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:06.694564    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1166de44_05d3_4773_b5ed_7e19eedaee71.slice/crio-862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097 WatchSource:0}: Error finding container 862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097: Status 404 returned error can't find the container with id 862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097
Oct 30 21:15:06.696000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:06.695975    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"]
Oct 30 21:15:07.555295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:07.555258    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6" event={"ID":"1166de44-05d3-4773-b5ed-7e19eedaee71","Type":"ContainerStarted","Data":"5596bd61c22402f12b6fd006aab3dfef6e6e4ee318e88ac296b401cbfdacf76b"}
Oct 30 21:15:07.555824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:07.555326    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6" event={"ID":"1166de44-05d3-4773-b5ed-7e19eedaee71","Type":"ContainerStarted","Data":"862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097"}
Oct 30 21:15:07.596724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:07.596674    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6" podStartSLOduration=1.5966582 podStartE2EDuration="1.5966582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:07.595850545 +0000 UTC m=+851.007389981" watchObservedRunningTime="2026-10-30 21:15:07.5966582 +0000 UTC m=+851.008197638"
Oct 30 21:15:08.564843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:08.564807    7037 generic.go:356] "Generic (PLEG): container finished" podID="1166de44-05d3-4773-b5ed-7e19eedaee71" containerID="5596bd61c22402f12b6fd006aab3dfef6e6e4ee318e88ac296b401cbfdacf76b" exitCode=0
Oct 30 21:15:08.565264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:08.564887    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6" event={"ID":"1166de44-05d3-4773-b5ed-7e19eedaee71","Type":"ContainerDied","Data":"5596bd61c22402f12b6fd006aab3dfef6e6e4ee318e88ac296b401cbfdacf76b"}
Oct 30 21:15:10.434638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.434587    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:10.499878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.499832    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:10.502224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.500596    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:10.659537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.654740    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:15:10.659537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.655134    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:10.942516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.926702    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:10.991227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:10.989169    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.078395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.046392    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.078395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.071147    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.294570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.294018    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:11.379529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.364507    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\") pod \"1166de44-05d3-4773-b5ed-7e19eedaee71\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") "
Oct 30 21:15:11.379529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.368750    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\" (UniqueName: \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\") pod \"1166de44-05d3-4773-b5ed-7e19eedaee71\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") "
Oct 30 21:15:11.379529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.368798    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\") pod \"1166de44-05d3-4773-b5ed-7e19eedaee71\" (UID: \"1166de44-05d3-4773-b5ed-7e19eedaee71\") "
Oct 30 21:15:11.449517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.382738    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca" pod "1166de44-05d3-4773-b5ed-7e19eedaee71" (UID: "1166de44-05d3-4773-b5ed-7e19eedaee71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:11.449517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.435599    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.449517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.436289    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:15:11.449517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.440731    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd" pod "1166de44-05d3-4773-b5ed-7e19eedaee71" (UID: "1166de44-05d3-4773-b5ed-7e19eedaee71"). InnerVolumeSpecName "kube-api-access-swwnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:11.449517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.441122    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca" pod "1166de44-05d3-4773-b5ed-7e19eedaee71" (UID: "1166de44-05d3-4773-b5ed-7e19eedaee71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.513631    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.513994    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.514418    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.514475    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.546353    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.546379    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1166de44-05d3-4773-b5ed-7e19eedaee71-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.546393    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwnd\" (UniqueName: \"kubernetes.io/projected/1166de44-05d3-4773-b5ed-7e19eedaee71-kube-api-access-swwnd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:11.560575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.557224    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.664205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.664162    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.703892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.687624    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:11.735133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.720573    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:15:11.735133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.720641    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:15:11.958517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.957067    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.000274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:11.979147    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:15:12.035632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.000611    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.035632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.000977    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:15:12.052574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.051374    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.052574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.051785    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:15:12.073697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.073654    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.080521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.074252    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:15:12.380572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.363985    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6" event={"ID":"1166de44-05d3-4773-b5ed-7e19eedaee71","Type":"ContainerDied","Data":"862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097"}
Oct 30 21:15:12.380572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.364050    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862b3197192b1dc9235a80d48ee6f2da46e252cd646c399b5e4e4613f7697097"
Oct 30 21:15:12.380572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.364124    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"
Oct 30 21:15:12.440763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.440344    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.514523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.514240    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.515034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.514664    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:12.585631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.585587    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.743610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.730750    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.743610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.731518    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:12.759328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:12.759279    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:13.043689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.043577    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.095780    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.096239    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.115473    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.115744    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.115808    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.156770    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.195383    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:15:13.200360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.195469    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:15:13.229750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.229709    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:15:13.240408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.240367    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:15:13.304423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.304176    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:15:13.306170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.305872    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:15:13.965843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.949835    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"]
Oct 30 21:15:13.965843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.950953    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1166de44-05d3-4773-b5ed-7e19eedaee71" containerName="revoke-keys"
Oct 30 21:15:13.965843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.950976    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1166de44-05d3-4773-b5ed-7e19eedaee71" containerName="revoke-keys"
Oct 30 21:15:13.965843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.951146    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1166de44-05d3-4773-b5ed-7e19eedaee71" containerName="revoke-keys"
Oct 30 21:15:13.968419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.967551    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"]
Oct 30 21:15:13.968419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.967673    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:13.982387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:13.982356    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:14.075673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.075624    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.075845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.075766    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8ff\" (UniqueName: \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.076029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.076005    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.179160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.177886    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.179160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.177993    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8ff\" (UniqueName: \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.179160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.178191    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.179160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.178806    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.181893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.181861    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.198894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.198848    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8ff\" (UniqueName: \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\") pod \"maas-api-revoke-sub-64816519-2bg66\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.285737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.285650    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:14.619826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:14.619794    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"]
Oct 30 21:15:14.622578 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:14.622537    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8df5461_967b_430a_8ba4_f91c33b97f00.slice/crio-bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f WatchSource:0}: Error finding container bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f: Status 404 returned error can't find the container with id bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f
Oct 30 21:15:15.398394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.398354    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66" event={"ID":"d8df5461-967b-430a-8ba4-f91c33b97f00","Type":"ContainerStarted","Data":"f611868bbe59533f4f1d288a732cb2fdaf1f07256005dd1df17e9843a1cf5dd6"}
Oct 30 21:15:15.398817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.398423    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66" event={"ID":"d8df5461-967b-430a-8ba4-f91c33b97f00","Type":"ContainerStarted","Data":"bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f"}
Oct 30 21:15:15.445270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.445222    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66" podStartSLOduration=2.445209563 podStartE2EDuration="2.445209563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:15.441979915 +0000 UTC m=+858.853519363" watchObservedRunningTime="2026-10-30 21:15:15.445209563 +0000 UTC m=+858.856748999"
Oct 30 21:15:15.669013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.668934    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:15:15.674765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.674742    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.700111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.700089    7037 scope.go:122] "RemoveContainer" containerID="cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"
Oct 30 21:15:15.706217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.706191    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707095    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707138    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707179    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707203    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707284    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707330    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707420    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.707889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.707513    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.712802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.712780    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:15:15.745984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.745950    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.782446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.782416    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:15:15.808620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808586    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808665    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808698    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808744    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808778    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808920    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254sm\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.808991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808960    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.808992    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809035    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809059    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809126    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.809167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809162    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.809243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809195    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809231    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.809356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809267    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809335    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809367    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809382    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809404    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.809770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.809746    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.810076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.810044    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.810076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.810048    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.810260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.810233    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.811767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.811742    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.812206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.812178    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.838775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.838745    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.838892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.838871    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:15.911410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911411    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911439    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911469    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911579    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254sm\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911682    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911710    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911751    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.911942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911881    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.912091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.911947    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.912147 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.912100    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.912232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.912209    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.912296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.912265    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.912354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.912300    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.913858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.913833    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.913959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.913930    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.934595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.934536    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.936141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.936121    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254sm\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:15.990336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:15.990309    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:16.065390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.065351    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:16.377371 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:16.377335    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473d9566_a4af_4747_8ca4_25c9b33d7e63.slice/crio-834d17c1fee3e7b72ac3f9d22bd3f653b1b4abd29a1cc9a8b6ce39021f829ef6 WatchSource:0}: Error finding container 834d17c1fee3e7b72ac3f9d22bd3f653b1b4abd29a1cc9a8b6ce39021f829ef6: Status 404 returned error can't find the container with id 834d17c1fee3e7b72ac3f9d22bd3f653b1b4abd29a1cc9a8b6ce39021f829ef6
Oct 30 21:15:16.378360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.378333    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:15:16.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.380281    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.380380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.380347    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.380380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.380377    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.408360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.408332    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerStarted","Data":"951801ab757f19b11223fa5fc8912c16f36c276b9907350137b0169dba71de3a"}
Oct 30 21:15:16.410156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.410129    7037 generic.go:356] "Generic (PLEG): container finished" podID="d8df5461-967b-430a-8ba4-f91c33b97f00" containerID="f611868bbe59533f4f1d288a732cb2fdaf1f07256005dd1df17e9843a1cf5dd6" exitCode=0
Oct 30 21:15:16.410265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.410184    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66" event={"ID":"d8df5461-967b-430a-8ba4-f91c33b97f00","Type":"ContainerDied","Data":"f611868bbe59533f4f1d288a732cb2fdaf1f07256005dd1df17e9843a1cf5dd6"}
Oct 30 21:15:16.412316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.412289    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" event={"ID":"473d9566-a4af-4747-8ca4-25c9b33d7e63","Type":"ContainerStarted","Data":"834d17c1fee3e7b72ac3f9d22bd3f653b1b4abd29a1cc9a8b6ce39021f829ef6"}
Oct 30 21:15:16.419328 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:16.419299    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d64bcd_2ca7_4936_a7a4_bf83c3cebb6f.slice/crio-0af017dfbbba8c6924fb176a7d7d53aedc3284104128a3f315411bfeb7444dca WatchSource:0}: Error finding container 0af017dfbbba8c6924fb176a7d7d53aedc3284104128a3f315411bfeb7444dca: Status 404 returned error can't find the container with id 0af017dfbbba8c6924fb176a7d7d53aedc3284104128a3f315411bfeb7444dca
Oct 30 21:15:16.420764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.420736    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:15:16.421929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.421899    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.422001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.421974    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.422041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.422005    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:16.574520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.573818    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:15:16.584544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.582216    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.622705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.622675    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:15:16.624598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624472    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.624698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624607    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.624698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624656    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.624780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624715    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.625336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624811    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599tz\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.625336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.624861    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.625336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.625023    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.625336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.625075    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.625336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.625149    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.706602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.706521    7037 scope.go:122] "RemoveContainer" containerID="f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"
Oct 30 21:15:16.707925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.707758    7037 scope.go:122] "RemoveContainer" containerID="2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"
Oct 30 21:15:16.726309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726273    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726566    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726654    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726680    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726855    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599tz\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726923    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.726997    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727034    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727093    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727182    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.727865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727650    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.728515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.727888    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.728988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.728962    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.730594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.729730    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.733684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.733660    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.749328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.749306    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.749413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.749373    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599tz\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:16.919325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:16.919290    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:17.307876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.306047    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:15:17.314896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.314858    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:17.314990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.314921    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:17.314990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.314948    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:17.493604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.473204    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerStarted","Data":"89df694821e58ef9ec5ce2b7a3c7794828e197ec542805871cc0675eab6e2848"}
Oct 30 21:15:17.518436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.517064    7037 generic.go:356] "Generic (PLEG): container finished" podID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerID="951801ab757f19b11223fa5fc8912c16f36c276b9907350137b0169dba71de3a" exitCode=60
Oct 30 21:15:17.518436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.517165    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerDied","Data":"951801ab757f19b11223fa5fc8912c16f36c276b9907350137b0169dba71de3a"}
Oct 30 21:15:17.518436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.517228    7037 scope.go:122] "RemoveContainer" containerID="cf49e369f9f5306e6fc6d0781752b798ae172c36527c99332c3679c7996f4957"
Oct 30 21:15:17.518436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.517745    7037 scope.go:122] "RemoveContainer" containerID="951801ab757f19b11223fa5fc8912c16f36c276b9907350137b0169dba71de3a"
Oct 30 21:15:17.518436 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:17.518084    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj_odh-ai-gateway-infra(95377ae4-1c7a-4681-a8a1-28b6062cb5ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac"
Oct 30 21:15:17.546561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.523561    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerStarted","Data":"6bf856c0e4c594b4cf341307ed3bd5bae47dfd224f74ac4b633b2452d3a64fbc"}
Oct 30 21:15:17.558668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.558561    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" event={"ID":"473d9566-a4af-4747-8ca4-25c9b33d7e63","Type":"ContainerStarted","Data":"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"}
Oct 30 21:15:17.561962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.561888    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:17.563100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.563001    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" event={"ID":"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f","Type":"ContainerStarted","Data":"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"}
Oct 30 21:15:17.563100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.563069    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" event={"ID":"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f","Type":"ContainerStarted","Data":"0af017dfbbba8c6924fb176a7d7d53aedc3284104128a3f315411bfeb7444dca"}
Oct 30 21:15:17.566334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.566281    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" event={"ID":"be596f6c-0bd4-4ae7-9075-1040c449f54d","Type":"ContainerStarted","Data":"45f6042ad681c2fb3036e30418b26b066712dc21ab241b67f81359d06812937a"}
Oct 30 21:15:17.701521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.701314    7037 scope.go:122] "RemoveContainer" containerID="e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"
Oct 30 21:15:17.912804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.912683    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podStartSLOduration=2.912664931 podStartE2EDuration="2.912664931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:17.813161151 +0000 UTC m=+861.224700597" watchObservedRunningTime="2026-10-30 21:15:17.912664931 +0000 UTC m=+861.324204369"
Oct 30 21:15:17.919654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:17.919623    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"]
Oct 30 21:15:18.014080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.014045    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:18.014750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.014716    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:18.036531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.020791    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podStartSLOduration=3.020774203 podStartE2EDuration="3.020774203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:17.94725101 +0000 UTC m=+861.358790459" watchObservedRunningTime="2026-10-30 21:15:18.020774203 +0000 UTC m=+861.432313641"
Oct 30 21:15:18.063514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.041774    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"]
Oct 30 21:15:18.080019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.079983    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:18.107357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.080275    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:18.111845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.049955    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.127000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.119665    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:18.127000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.119714    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:18.127000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.123302    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8n9\" (UniqueName: \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.158783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.145666    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.203571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.200170    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.317553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.312273    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8n9\" (UniqueName: \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.317553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.312467    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.317553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.312631    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.317553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.313680    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.346136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.346106    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:18.366075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.366002    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.395451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.395419    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8n9\" (UniqueName: \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\") pod \"maas-api-revoke-sub-a059beda-fbx8b\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.537517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.522236    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\") pod \"d8df5461-967b-430a-8ba4-f91c33b97f00\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") "
Oct 30 21:15:18.537517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.522314    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\" (UniqueName: \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\") pod \"d8df5461-967b-430a-8ba4-f91c33b97f00\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") "
Oct 30 21:15:18.537517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.522590    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\") pod \"d8df5461-967b-430a-8ba4-f91c33b97f00\" (UID: \"d8df5461-967b-430a-8ba4-f91c33b97f00\") "
Oct 30 21:15:18.537517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.524025    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca" pod "d8df5461-967b-430a-8ba4-f91c33b97f00" (UID: "d8df5461-967b-430a-8ba4-f91c33b97f00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.593239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.584617    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca" pod "d8df5461-967b-430a-8ba4-f91c33b97f00" (UID: "d8df5461-967b-430a-8ba4-f91c33b97f00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:18.600600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.600394    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:18.607316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.607258    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff" pod "d8df5461-967b-430a-8ba4-f91c33b97f00" (UID: "d8df5461-967b-430a-8ba4-f91c33b97f00"). InnerVolumeSpecName "kube-api-access-bn8ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:18.612623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.611771    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerStarted","Data":"813ed24b30c5bda9043cb0d34ae2bd64f3a5cbcaa801573297d3f6c22115d876"}
Oct 30 21:15:18.614618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.614585    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" event={"ID":"be596f6c-0bd4-4ae7-9075-1040c449f54d","Type":"ContainerStarted","Data":"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"}
Oct 30 21:15:18.622937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.622787    7037 generic.go:356] "Generic (PLEG): container finished" podID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerID="89df694821e58ef9ec5ce2b7a3c7794828e197ec542805871cc0675eab6e2848" exitCode=60
Oct 30 21:15:18.623299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.623081    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerDied","Data":"89df694821e58ef9ec5ce2b7a3c7794828e197ec542805871cc0675eab6e2848"}
Oct 30 21:15:18.623299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.623188    7037 scope.go:122] "RemoveContainer" containerID="f68a0f2d4a662a57939641926e1dadd550441ccad721a6d386fe86fe6caf3441"
Oct 30 21:15:18.632804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.625742    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:18.632804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.625779    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8df5461-967b-430a-8ba4-f91c33b97f00-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:18.632804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.625796    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8ff\" (UniqueName: \"kubernetes.io/projected/d8df5461-967b-430a-8ba4-f91c33b97f00-kube-api-access-bn8ff\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:18.655717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.650767    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:18.700470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.700392    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:18.718284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.718243    7037 generic.go:356] "Generic (PLEG): container finished" podID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerID="6bf856c0e4c594b4cf341307ed3bd5bae47dfd224f74ac4b633b2452d3a64fbc" exitCode=60
Oct 30 21:15:18.746798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.723450    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podStartSLOduration=2.723430025 podStartE2EDuration="2.723430025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:18.714891694 +0000 UTC m=+862.126431134" watchObservedRunningTime="2026-10-30 21:15:18.723430025 +0000 UTC m=+862.134969462"
Oct 30 21:15:18.747567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.747530    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerDied","Data":"6bf856c0e4c594b4cf341307ed3bd5bae47dfd224f74ac4b633b2452d3a64fbc"}
Oct 30 21:15:18.790559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.790414    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"
Oct 30 21:15:18.790884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.790767    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66" event={"ID":"d8df5461-967b-430a-8ba4-f91c33b97f00","Type":"ContainerDied","Data":"bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f"}
Oct 30 21:15:18.790884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.790821    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bab4a8c0d8ea91d0756961296f2375c58691e6ed43c47fff335623ee117f024f"
Oct 30 21:15:18.791666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.791649    7037 scope.go:122] "RemoveContainer" containerID="2cc0a18dceb86327d8cab25fb8cebb7d5b2741654fc62e1fa3b5013e4ea30887"
Oct 30 21:15:18.923660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.923578    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:18.959542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.939827    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:18.962607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.962550    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:19.021590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.996212    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:19.021590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:18.996271    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:19.117747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.116415    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:19.117747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.116479    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:19.636515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.634837    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:19.717693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.717656    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\") pod \"ef02bca8-e10c-4474-bda7-de776ef80bdb\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") "
Oct 30 21:15:19.718063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.718041    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\" (UniqueName: \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\") pod \"ef02bca8-e10c-4474-bda7-de776ef80bdb\" (UID: \"ef02bca8-e10c-4474-bda7-de776ef80bdb\") "
Oct 30 21:15:19.719256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.719230    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca" pod "ef02bca8-e10c-4474-bda7-de776ef80bdb" (UID: "ef02bca8-e10c-4474-bda7-de776ef80bdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.747777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.747641    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:19.854687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.821261    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02bca8-e10c-4474-bda7-de776ef80bdb-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:19.905280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.903832    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk" pod "ef02bca8-e10c-4474-bda7-de776ef80bdb" (UID: "ef02bca8-e10c-4474-bda7-de776ef80bdb"). InnerVolumeSpecName "kube-api-access-zx9hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.916279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.915035    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd" event={"ID":"ef02bca8-e10c-4474-bda7-de776ef80bdb","Type":"ContainerDied","Data":"f8c6da429d31723578e9079b6c320e289827b02940f92068c766201a27aaad0c"}
Oct 30 21:15:19.916279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.915163    7037 scope.go:122] "RemoveContainer" containerID="89df694821e58ef9ec5ce2b7a3c7794828e197ec542805871cc0675eab6e2848"
Oct 30 21:15:19.916279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.915193    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"
Oct 30 21:15:19.922225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.922202    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:20.007616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.003165    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:20.007616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:19.924050    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9hk\" (UniqueName: \"kubernetes.io/projected/ef02bca8-e10c-4474-bda7-de776ef80bdb-kube-api-access-zx9hk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:20.008017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.007989    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:20.008170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.008146    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:20.016955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.016930    7037 generic.go:356] "Generic (PLEG): container finished" podID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerID="813ed24b30c5bda9043cb0d34ae2bd64f3a5cbcaa801573297d3f6c22115d876" exitCode=60
Oct 30 21:15:20.017053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.017025    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerDied","Data":"813ed24b30c5bda9043cb0d34ae2bd64f3a5cbcaa801573297d3f6c22115d876"}
Oct 30 21:15:20.020131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.020077    7037 scope.go:122] "RemoveContainer" containerID="e044c721b40dd1fb8fdd4180251bf6ca42b059aa0883058ea82ebccccec833b3"
Oct 30 21:15:20.065070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.064981    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:20.093522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.093387    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:20.093522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.093452    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:20.106522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.106440    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:20.254515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.218437    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\" (UniqueName: \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\") pod \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") "
Oct 30 21:15:20.254515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.218504    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\" (UniqueName: \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\") pod \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") "
Oct 30 21:15:20.254515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.218606    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\") pod \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\" (UID: \"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557\") "
Oct 30 21:15:20.254515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.218877    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\") pod \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\" (UID: \"95377ae4-1c7a-4681-a8a1-28b6062cb5ac\") "
Oct 30 21:15:20.254515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.250390    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca" pod "95377ae4-1c7a-4681-a8a1-28b6062cb5ac" (UID: "95377ae4-1c7a-4681-a8a1-28b6062cb5ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.298588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.297181    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca" pod "87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" (UID: "87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.336550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.335905    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:20.336793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.336697    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7" pod "87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" (UID: "87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557"). InnerVolumeSpecName "kube-api-access-kjrr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.409596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.363896    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrr7\" (UniqueName: \"kubernetes.io/projected/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-kube-api-access-kjrr7\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:20.409596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.363934    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:20.409596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.363953    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:20.409596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.404632    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6" pod "95377ae4-1c7a-4681-a8a1-28b6062cb5ac" (UID: "95377ae4-1c7a-4681-a8a1-28b6062cb5ac"). InnerVolumeSpecName "kube-api-access-54mq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.523197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.523163    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mq6\" (UniqueName: \"kubernetes.io/projected/95377ae4-1c7a-4681-a8a1-28b6062cb5ac-kube-api-access-54mq6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:20.540516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.536760    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-vzrnd"]
Oct 30 21:15:20.717717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:20.694247    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"]
Oct 30 21:15:21.027570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.007187    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:21.027570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.007247    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:21.027570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.008098    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:21.027570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.008155    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:21.109744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.083132    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:21.109744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.083202    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:21.409511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.358607    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:21.409511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.404699    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\" (UniqueName: \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\") pod \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") "
Oct 30 21:15:21.413391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.413349    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"
Oct 30 21:15:21.415184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.415157    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2" pod "9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" (UID: "9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44"). InnerVolumeSpecName "kube-api-access-5hcf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:21.518695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.518564    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\") pod \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\" (UID: \"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44\") "
Oct 30 21:15:21.557535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.520610    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"
Oct 30 21:15:21.557535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.523455    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hcf2\" (UniqueName: \"kubernetes.io/projected/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-kube-api-access-5hcf2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:21.595163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.595105    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca" pod "9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" (UID: "9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:21.623365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.623323    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" path="/var/lib/kubelet/pods/ef02bca8-e10c-4474-bda7-de776ef80bdb/volumes"
Oct 30 21:15:21.631523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.624651    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b" event={"ID":"228a0698-53eb-42da-9a1e-513b6f9673c9","Type":"ContainerStarted","Data":"f44159ddffd824d52d6f6bc1ae9b2cb841cdc02a9cc8deba1cd0e5b056dcac38"}
Oct 30 21:15:21.631523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.624727    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj" event={"ID":"95377ae4-1c7a-4681-a8a1-28b6062cb5ac","Type":"ContainerDied","Data":"aa084f68c2198562f801715dc8ed01c5e151785da16e9051cefa1d95e3355fdc"}
Oct 30 21:15:21.631523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.629743    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq" event={"ID":"87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557","Type":"ContainerDied","Data":"315ba57bab029e0f55c251e4a0efff03967229e950c53dcc1089b596ec1ce9c9"}
Oct 30 21:15:21.631523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.629907    7037 scope.go:122] "RemoveContainer" containerID="951801ab757f19b11223fa5fc8912c16f36c276b9907350137b0169dba71de3a"
Oct 30 21:15:21.746092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.721821    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:21.817061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.815309    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"
Oct 30 21:15:21.822541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.821911    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64" event={"ID":"9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44","Type":"ContainerDied","Data":"9a865f26bf80a0fd5b4815274e0b4d33c8227892197c7e7c60795d033910f4c3"}
Oct 30 21:15:21.959877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.959825    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:21.960070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:21.959901    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:22.008854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.008820    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:22.009099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.009073    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:22.091567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.091528    7037 scope.go:122] "RemoveContainer" containerID="6bf856c0e4c594b4cf341307ed3bd5bae47dfd224f74ac4b633b2452d3a64fbc"
Oct 30 21:15:22.122969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.122938    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:22.123656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.123620    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:22.321115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.321079    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:22.419448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.419379    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889915-6wbxq"]
Oct 30 21:15:22.444513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.429678    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:22.445227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.445200    7037 scope.go:122] "RemoveContainer" containerID="813ed24b30c5bda9043cb0d34ae2bd64f3a5cbcaa801573297d3f6c22115d876"
Oct 30 21:15:22.611541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.608546    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889915-wvx64"]
Oct 30 21:15:22.618130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.618099    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:22.623173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.623140    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889915-jf9gj"]
Oct 30 21:15:22.717330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.717239    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" path="/var/lib/kubelet/pods/87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557/volumes"
Oct 30 21:15:22.717976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.717955    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" path="/var/lib/kubelet/pods/95377ae4-1c7a-4681-a8a1-28b6062cb5ac/volumes"
Oct 30 21:15:22.718448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.718430    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" path="/var/lib/kubelet/pods/9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44/volumes"
Oct 30 21:15:22.858522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.858285    7037 generic.go:356] "Generic (PLEG): container finished" podID="228a0698-53eb-42da-9a1e-513b6f9673c9" containerID="f807e6134d463fadd3ab871b277098130e3bbc5c715d5ad64e480cc86d64990c" exitCode=0
Oct 30 21:15:22.859010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.858550    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b" event={"ID":"228a0698-53eb-42da-9a1e-513b6f9673c9","Type":"ContainerDied","Data":"f807e6134d463fadd3ab871b277098130e3bbc5c715d5ad64e480cc86d64990c"}
Oct 30 21:15:22.919923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.919887    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:22.920116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.919941    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:22.970221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970129    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"]
Oct 30 21:15:22.970831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970814    7037 cpu_manager.go:414] "containerMap: removing container" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970831    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970847    7037 cpu_manager.go:414] "containerMap: removing container" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970853    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970872    7037 cpu_manager.go:414] "containerMap: removing container" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970878    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970891    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970896    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970911    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d8df5461-967b-430a-8ba4-f91c33b97f00" containerName="revoke-keys"
Oct 30 21:15:22.970914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970915    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8df5461-967b-430a-8ba4-f91c33b97f00" containerName="revoke-keys"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970924    7037 cpu_manager.go:414] "containerMap: removing container" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970929    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970935    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970939    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970946    7037 cpu_manager.go:414] "containerMap: removing container" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970953    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970964    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970969    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970978    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970982    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.970999    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971003    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971114    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971122    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971129    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971134    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971141    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971147    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971154    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971165    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8df5461-967b-430a-8ba4-f91c33b97f00" containerName="revoke-keys"
Oct 30 21:15:22.971333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.971172    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:22.980258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.978930    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:22.985174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.982528    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"]
Oct 30 21:15:22.990724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.990694    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:22.990860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:22.990746    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:23.069260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.069227    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:23.069399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.069274    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:23.154529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.140530    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.154529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.140590    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.154529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.140852    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6b7l\" (UniqueName: \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.258571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.257226    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6b7l\" (UniqueName: \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.258571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.257459    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.258571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.257541    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.274965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.274927    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.285505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.281940    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.296283 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.296255    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6b7l\" (UniqueName: \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\") pod \"maas-api-revoke-sub-c87b0cf2-cdcfh\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.299163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.299133    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:23.758190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.757520    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"]
Oct 30 21:15:23.759904 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:23.759863    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2aa6ec_0950_4882_a906_f4c7afd67c94.slice/crio-6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e WatchSource:0}: Error finding container 6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e: Status 404 returned error can't find the container with id 6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e
Oct 30 21:15:23.916450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.916329    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh" event={"ID":"ae2aa6ec-0950-4882-a906-f4c7afd67c94","Type":"ContainerStarted","Data":"6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e"}
Oct 30 21:15:23.987522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.920978    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:23.987522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.921027    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:23.991327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.991291    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:15:23.991432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:23.991360    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 21:15:24.066916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.066310    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:15:24.066916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.066370    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 21:15:24.184384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.184353    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:24.308000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.307115    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\") pod \"228a0698-53eb-42da-9a1e-513b6f9673c9\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") "
Oct 30 21:15:24.308000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.307221    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\") pod \"228a0698-53eb-42da-9a1e-513b6f9673c9\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") "
Oct 30 21:15:24.308000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.307269    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\" (UniqueName: \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\") pod \"228a0698-53eb-42da-9a1e-513b6f9673c9\" (UID: \"228a0698-53eb-42da-9a1e-513b6f9673c9\") "
Oct 30 21:15:24.308863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.308836    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca" pod "228a0698-53eb-42da-9a1e-513b6f9673c9" (UID: "228a0698-53eb-42da-9a1e-513b6f9673c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:24.310452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.310431    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca" pod "228a0698-53eb-42da-9a1e-513b6f9673c9" (UID: "228a0698-53eb-42da-9a1e-513b6f9673c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:24.313979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.313951    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9" pod "228a0698-53eb-42da-9a1e-513b6f9673c9" (UID: "228a0698-53eb-42da-9a1e-513b6f9673c9"). InnerVolumeSpecName "kube-api-access-ml8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:24.409887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.409849    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:24.409887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.409885    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/228a0698-53eb-42da-9a1e-513b6f9673c9-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:24.410138 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.409899    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8n9\" (UniqueName: \"kubernetes.io/projected/228a0698-53eb-42da-9a1e-513b6f9673c9-kube-api-access-ml8n9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:24.920484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.920449    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:15:24.920974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.920528    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:15:24.927800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.927775    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"
Oct 30 21:15:24.927958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.927775    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b" event={"ID":"228a0698-53eb-42da-9a1e-513b6f9673c9","Type":"ContainerDied","Data":"f44159ddffd824d52d6f6bc1ae9b2cb841cdc02a9cc8deba1cd0e5b056dcac38"}
Oct 30 21:15:24.927958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.927927    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f44159ddffd824d52d6f6bc1ae9b2cb841cdc02a9cc8deba1cd0e5b056dcac38"
Oct 30 21:15:24.929812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.929776    7037 generic.go:356] "Generic (PLEG): container finished" podID="ae2aa6ec-0950-4882-a906-f4c7afd67c94" containerID="2a99344824c20a29d9460df03b73078f417edea9b018780bf2a80b9cec7f0334" exitCode=0
Oct 30 21:15:24.929965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.929944    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh" event={"ID":"ae2aa6ec-0950-4882-a906-f4c7afd67c94","Type":"ContainerDied","Data":"2a99344824c20a29d9460df03b73078f417edea9b018780bf2a80b9cec7f0334"}
Oct 30 21:15:24.995835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.995755    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:24.996205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.996181    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:24.996907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:24.996886    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:15:25.075009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:25.074974    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:25.075474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:25.075442    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:25.076543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:25.076483    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:15:26.156022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.155990    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:26.157699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.157674    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:26.161126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.161103    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:15:26.162611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.162562    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:26.242831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.242752    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\" (UniqueName: \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\") pod \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") "
Oct 30 21:15:26.243022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.243001    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\") pod \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") "
Oct 30 21:15:26.243188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.243168    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\") pod \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\" (UID: \"ae2aa6ec-0950-4882-a906-f4c7afd67c94\") "
Oct 30 21:15:26.243592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.243551    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca" pod "ae2aa6ec-0950-4882-a906-f4c7afd67c94" (UID: "ae2aa6ec-0950-4882-a906-f4c7afd67c94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:26.243937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.243912    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.249335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.249305    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca" pod "ae2aa6ec-0950-4882-a906-f4c7afd67c94" (UID: "ae2aa6ec-0950-4882-a906-f4c7afd67c94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:26.256183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.256151    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l" pod "ae2aa6ec-0950-4882-a906-f4c7afd67c94" (UID: "ae2aa6ec-0950-4882-a906-f4c7afd67c94"). InnerVolumeSpecName "kube-api-access-c6b7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:26.345168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.345135    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6b7l\" (UniqueName: \"kubernetes.io/projected/ae2aa6ec-0950-4882-a906-f4c7afd67c94-kube-api-access-c6b7l\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.345168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.345167    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2aa6ec-0950-4882-a906-f4c7afd67c94-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.949668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.949632    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"
Oct 30 21:15:26.949864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.949663    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh" event={"ID":"ae2aa6ec-0950-4882-a906-f4c7afd67c94","Type":"ContainerDied","Data":"6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e"}
Oct 30 21:15:26.949864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:26.949721    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c1e4b1882a48ede4787840d07141db774a2ac2f68d163b9eed99b25035f394e"
Oct 30 21:15:28.265372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.265341    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"]
Oct 30 21:15:28.266095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266077    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ae2aa6ec-0950-4882-a906-f4c7afd67c94" containerName="revoke-keys"
Oct 30 21:15:28.266144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266097    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2aa6ec-0950-4882-a906-f4c7afd67c94" containerName="revoke-keys"
Oct 30 21:15:28.266144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266110    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:28.266144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266115    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:28.266144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266132    7037 cpu_manager.go:414] "containerMap: removing container" podUID="228a0698-53eb-42da-9a1e-513b6f9673c9" containerName="revoke-keys"
Oct 30 21:15:28.266144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266137    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228a0698-53eb-42da-9a1e-513b6f9673c9" containerName="revoke-keys"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266161    7037 cpu_manager.go:414] "containerMap: removing container" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266167    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266254    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d0eeb7-5cf4-4f3d-af7e-f6918f7e1557" containerName="cleanup"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266260    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="228a0698-53eb-42da-9a1e-513b6f9673c9" containerName="revoke-keys"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266267    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="95377ae4-1c7a-4681-a8a1-28b6062cb5ac" containerName="cleanup"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266280    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2aa6ec-0950-4882-a906-f4c7afd67c94" containerName="revoke-keys"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266288    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2cf21a-c93f-4eb7-8d3f-0168f8cd4d44" containerName="cleanup"
Oct 30 21:15:28.266383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.266296    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef02bca8-e10c-4474-bda7-de776ef80bdb" containerName="cleanup"
Oct 30 21:15:28.270167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.270151    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.289073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.289047    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"]
Oct 30 21:15:28.295449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.295422    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:28.295586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.295463    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:15:28.373545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.373484    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.373715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.373635    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.373776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.373761    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7lz\" (UniqueName: \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.475207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.475153    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.475416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.475251    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.475416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.475373    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7lz\" (UniqueName: \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.476010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.475983    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.478114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.478088    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.495011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.494986    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7lz\" (UniqueName: \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\") pod \"maas-api-revoke-sub-7facfa59-9nbdg\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.588676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.588646    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:28.883506 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:28.883459    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927d44d3_7995_439a_99ab_601e46672b93.slice/crio-71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50 WatchSource:0}: Error finding container 71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50: Status 404 returned error can't find the container with id 71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50
Oct 30 21:15:28.885504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.885469    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"]
Oct 30 21:15:28.967319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:28.967287    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg" event={"ID":"927d44d3-7995-439a-99ab-601e46672b93","Type":"ContainerStarted","Data":"71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50"}
Oct 30 21:15:29.977940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:29.977852    7037 generic.go:356] "Generic (PLEG): container finished" podID="927d44d3-7995-439a-99ab-601e46672b93" containerID="5cc2b25420f27a5dddfbbcfad7ce3ff664a5f651c40e70902b65e14a8cb0f46b" exitCode=0
Oct 30 21:15:29.977940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:29.977917    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg" event={"ID":"927d44d3-7995-439a-99ab-601e46672b93","Type":"ContainerDied","Data":"5cc2b25420f27a5dddfbbcfad7ce3ff664a5f651c40e70902b65e14a8cb0f46b"}
Oct 30 21:15:30.609249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.609215    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:15:30.613876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.613854    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.650098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.650065    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:15:30.709817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.709783    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.709817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.709819    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.709852    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710005    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jttx\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710033    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710079    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710097    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710154    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.710237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.710175    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.811789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.811756    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.811789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.811792    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.811822    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.811989    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jttx\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812028    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812215    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812286    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812334    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812359    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812480    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812545    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812645    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812669    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.812877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.812712    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.814534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.814512    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.815179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.815154    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.835784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.835756    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jttx\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.835927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.835900    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:30.928673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:30.928597    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:31.184638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.184572    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:31.219680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.219648    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\") pod \"927d44d3-7995-439a-99ab-601e46672b93\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") "
Oct 30 21:15:31.219850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.219803    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\" (UniqueName: \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\") pod \"927d44d3-7995-439a-99ab-601e46672b93\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") "
Oct 30 21:15:31.220286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.220235    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\") pod \"927d44d3-7995-439a-99ab-601e46672b93\" (UID: \"927d44d3-7995-439a-99ab-601e46672b93\") "
Oct 30 21:15:31.221313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.221287    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca" pod "927d44d3-7995-439a-99ab-601e46672b93" (UID: "927d44d3-7995-439a-99ab-601e46672b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:31.221961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.221924    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/927d44d3-7995-439a-99ab-601e46672b93-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:31.222917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.222887    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz" pod "927d44d3-7995-439a-99ab-601e46672b93" (UID: "927d44d3-7995-439a-99ab-601e46672b93"). InnerVolumeSpecName "kube-api-access-mj7lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:31.223042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.222919    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca" pod "927d44d3-7995-439a-99ab-601e46672b93" (UID: "927d44d3-7995-439a-99ab-601e46672b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:31.249170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.249136    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:15:31.251963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.251921    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:31.252058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.252005    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:31.252058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.252046    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:31.323405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.323372    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/927d44d3-7995-439a-99ab-601e46672b93-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:31.323529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:31.323412    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7lz\" (UniqueName: \"kubernetes.io/projected/927d44d3-7995-439a-99ab-601e46672b93-kube-api-access-mj7lz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:32.003579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.003550    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"
Oct 30 21:15:32.003727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.003585    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg" event={"ID":"927d44d3-7995-439a-99ab-601e46672b93","Type":"ContainerDied","Data":"71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50"}
Oct 30 21:15:32.003727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.003632    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ae931244aae9ac239c858d634797aa0cebc73ff7462617b09cab7d8306fb50"
Oct 30 21:15:32.005392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.005369    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" event={"ID":"d3aeb557-2e9d-404a-bfb1-be2b127efc10","Type":"ContainerStarted","Data":"d339c5aa135d7814fa1a8c59ba6d06f4839920dfbe031b52be0fd8157b28b65d"}
Oct 30 21:15:32.005515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.005406    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" event={"ID":"d3aeb557-2e9d-404a-bfb1-be2b127efc10","Type":"ContainerStarted","Data":"1d4f5cd45361b43abd6c6ae18c2779d66309d35bdc840d1f6b3d416c9f7f71e0"}
Oct 30 21:15:32.066074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.066018    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podStartSLOduration=2.066004439 podStartE2EDuration="2.066004439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:32.063174317 +0000 UTC m=+875.474713754" watchObservedRunningTime="2026-10-30 21:15:32.066004439 +0000 UTC m=+875.477543876"
Oct 30 21:15:32.946512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.942547    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:32.949840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.949356    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.949840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:32.949446    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:33.675987 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:33.675948    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:15:33.944574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:33.931900    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.944574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:33.931963    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:34.929521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:34.929097    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.929521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:34.929159    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:35.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:35.935582    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:35.935645    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:36.687720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.687454    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"]
Oct 30 21:15:36.688621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.688595    7037 cpu_manager.go:414] "containerMap: removing container" podUID="927d44d3-7995-439a-99ab-601e46672b93" containerName="revoke-keys"
Oct 30 21:15:36.688831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.688807    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927d44d3-7995-439a-99ab-601e46672b93" containerName="revoke-keys"
Oct 30 21:15:36.689162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.689144    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="927d44d3-7995-439a-99ab-601e46672b93" containerName="revoke-keys"
Oct 30 21:15:36.695084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.695067    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.720597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.720474    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"]
Oct 30 21:15:36.721247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.721218    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:36.766557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.742732    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:15:36.856701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.849748    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2w2\" (UniqueName: \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.860512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.858739    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.860512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.859052    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.949009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.948910    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.949009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.948973    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:36.960796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.960765    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2w2\" (UniqueName: \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.960937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.960914    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.961153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.961134    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.961784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.961762    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.963250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.963230    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:36.986000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:36.985978    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2w2\" (UniqueName: \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\") pod \"maas-api-revoke-sub-108f89c6-pf92p\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:37.016963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.016933    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:37.350751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.350703    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"]
Oct 30 21:15:37.356744 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:37.356710    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2bbf29_b9e4_4894_a351_b210f0b6b18e.slice/crio-befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a WatchSource:0}: Error finding container befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a: Status 404 returned error can't find the container with id befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a
Oct 30 21:15:37.900120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.900092    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"]
Oct 30 21:15:37.904640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.904621    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:37.925119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.925085    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"]
Oct 30 21:15:37.929141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.929114    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.929312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:37.929155    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:15:38.081452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.081420    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.081904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.081696    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.081904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.081745    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9d6\" (UniqueName: \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.083465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.083440    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p" event={"ID":"1e2bbf29-b9e4-4894-a351-b210f0b6b18e","Type":"ContainerStarted","Data":"507f1458599240f2e38ddd72bb5c1135e3b7b5b793e33b786303b33eb8988435"}
Oct 30 21:15:38.083606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.083508    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p" event={"ID":"1e2bbf29-b9e4-4894-a351-b210f0b6b18e","Type":"ContainerStarted","Data":"befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a"}
Oct 30 21:15:38.131310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.131249    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p" podStartSLOduration=2.131229413 podStartE2EDuration="2.131229413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:38.125569825 +0000 UTC m=+881.537109263" watchObservedRunningTime="2026-10-30 21:15:38.131229413 +0000 UTC m=+881.542768851"
Oct 30 21:15:38.182982 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.182940    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.183182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.183134    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.183182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.183178    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9d6\" (UniqueName: \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.183870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.183845    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.185420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.185396    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.204591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.204550    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9d6\" (UniqueName: \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\") pod \"maas-api-revoke-sub-12b069a2-kl4k7\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.219649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.219621    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:38.539900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.539812    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"]
Oct 30 21:15:38.551949 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:38.551811    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f3637_5828_4d8f_845f_ae9a0f2170c8.slice/crio-9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873 WatchSource:0}: Error finding container 9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873: Status 404 returned error can't find the container with id 9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873
Oct 30 21:15:38.934126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.934097    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:38.934469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.934450    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:38.935235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:38.935218    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:15:39.095251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:39.095167    7037 generic.go:356] "Generic (PLEG): container finished" podID="1e2bbf29-b9e4-4894-a351-b210f0b6b18e" containerID="507f1458599240f2e38ddd72bb5c1135e3b7b5b793e33b786303b33eb8988435" exitCode=0
Oct 30 21:15:39.095709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:39.095256    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p" event={"ID":"1e2bbf29-b9e4-4894-a351-b210f0b6b18e","Type":"ContainerDied","Data":"507f1458599240f2e38ddd72bb5c1135e3b7b5b793e33b786303b33eb8988435"}
Oct 30 21:15:39.097320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:39.097293    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7" event={"ID":"216f3637-5828-4d8f-845f-ae9a0f2170c8","Type":"ContainerStarted","Data":"bf878934b9251bccc75099da97daae3cc22f62bfeaee07c8f05f98729fdd1ef8"}
Oct 30 21:15:39.097433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:39.097340    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7" event={"ID":"216f3637-5828-4d8f-845f-ae9a0f2170c8","Type":"ContainerStarted","Data":"9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873"}
Oct 30 21:15:39.177554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:39.177463    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7" podStartSLOduration=2.177450223 podStartE2EDuration="2.177450223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:39.173737654 +0000 UTC m=+882.585277090" watchObservedRunningTime="2026-10-30 21:15:39.177450223 +0000 UTC m=+882.588989659"
Oct 30 21:15:40.166943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:40.166907    7037 generic.go:356] "Generic (PLEG): container finished" podID="216f3637-5828-4d8f-845f-ae9a0f2170c8" containerID="bf878934b9251bccc75099da97daae3cc22f62bfeaee07c8f05f98729fdd1ef8" exitCode=0
Oct 30 21:15:40.172543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:40.171535    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7" event={"ID":"216f3637-5828-4d8f-845f-ae9a0f2170c8","Type":"ContainerDied","Data":"bf878934b9251bccc75099da97daae3cc22f62bfeaee07c8f05f98729fdd1ef8"}
Oct 30 21:15:40.957553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:40.936916    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:41.168516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.165598    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:41.243906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.202857    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:41.243906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.203257    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:15:41.243906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.209375    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:41.243906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.211752    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:15:41.279627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.279576    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:41.322183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.280550    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:41.586980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.586926    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:15:41.626524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.605448    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 21:15:41.968575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:41.945397    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.083960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.083055    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:15:42.247575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.201248    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.247575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.202135    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:15:42.247575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.242569    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.288528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.275237    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.521204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.520007    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:15:42.521204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.520095    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:15:42.557672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.557627    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.676758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.557854    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.693597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.693541    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:15:42.693921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.693906    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:15:42.756518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.588775    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 30 21:15:42.805127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.592635    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.806687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.557916    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:42.806888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.806859    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.807056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.559189    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.808255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.808235    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:15:42.812510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.808411    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:15:42.904731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.904666    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.923739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.923698    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 21:15:42.924116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:42.924081    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:15:43.032539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.001837    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:43.095658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.095610    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.174779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.174720    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.250522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.213749    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:43.270514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.258689    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.294423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.284875    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\") pod \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") "
Oct 30 21:15:43.294423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.284948    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\" (UniqueName: \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\") pod \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") "
Oct 30 21:15:43.310636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.310582    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.326804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.326765    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca" pod "1e2bbf29-b9e4-4894-a351-b210f0b6b18e" (UID: "1e2bbf29-b9e4-4894-a351-b210f0b6b18e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:43.443580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.438833    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\") pod \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\" (UID: \"1e2bbf29-b9e4-4894-a351-b210f0b6b18e\") "
Oct 30 21:15:43.520689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.520641    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:43.626719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.626658    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.692679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.692630    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.817972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.817934    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:15:43.853526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.818229    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2" pod "1e2bbf29-b9e4-4894-a351-b210f0b6b18e" (UID: "1e2bbf29-b9e4-4894-a351-b210f0b6b18e"). InnerVolumeSpecName "kube-api-access-5p2w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:43.853526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.818299    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.985685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.937831    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca" pod "1e2bbf29-b9e4-4894-a351-b210f0b6b18e" (UID: "1e2bbf29-b9e4-4894-a351-b210f0b6b18e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:44.076814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.958383    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2w2\" (UniqueName: \"kubernetes.io/projected/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-kube-api-access-5p2w2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:44.110254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:43.958758    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.111455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.105098    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p" event={"ID":"1e2bbf29-b9e4-4894-a351-b210f0b6b18e","Type":"ContainerDied","Data":"befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a"}
Oct 30 21:15:44.111691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.111669    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befbf06128a57900b96b61ffaf71452d581facf836cf216953833ccce961286a"
Oct 30 21:15:44.111793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.105217    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"
Oct 30 21:15:44.241516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.240715    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.253516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.244847    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.274604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.274553    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.341200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.341097    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.343424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.343394    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e2bbf29-b9e4-4894-a351-b210f0b6b18e-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:44.648529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.640518    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.745516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.739744    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.887574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.882589    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.932514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:44.928639    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.262571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.239819    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.320722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.320674    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.323078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.323042    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.359589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.355853    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.364520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.360718    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.667433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.667403    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:45.669236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.646819    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.782520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.770332    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.937868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.933918    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.937868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:45.934059    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.293602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.293472    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.367615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.367580    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\") pod \"216f3637-5828-4d8f-845f-ae9a0f2170c8\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") "
Oct 30 21:15:46.446279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.446246    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\" (UniqueName: \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\") pod \"216f3637-5828-4d8f-845f-ae9a0f2170c8\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") "
Oct 30 21:15:46.446617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.446590    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\") pod \"216f3637-5828-4d8f-845f-ae9a0f2170c8\" (UID: \"216f3637-5828-4d8f-845f-ae9a0f2170c8\") "
Oct 30 21:15:46.447653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.394700    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.448536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.394646    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.560661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.396353    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.561375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.370602    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.561855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.401968    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca" pod "216f3637-5828-4d8f-845f-ae9a0f2170c8" (UID: "216f3637-5828-4d8f-845f-ae9a0f2170c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:46.677670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.674669    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.772705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.772112    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.772705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.772368    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:46.825095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.823969    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2" podUID="5e30ce33-adeb-4935-91db-42bf140e78fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.854898 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:46.850259    7037 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.145s"
Oct 30 21:15:46.854898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.850399    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7" event={"ID":"216f3637-5828-4d8f-845f-ae9a0f2170c8","Type":"ContainerDied","Data":"9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873"}
Oct 30 21:15:46.854898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.850432    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb9e3917ec1c251bdcd706f4f907a93f2755bf8b43e697c2b805940d0283873"
Oct 30 21:15:46.881198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.881168    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"
Oct 30 21:15:46.945953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.945908    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.975794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:46.968765    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.047962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.047926    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6" pod "216f3637-5828-4d8f-845f-ae9a0f2170c8" (UID: "216f3637-5828-4d8f-845f-ae9a0f2170c8"). InnerVolumeSpecName "kube-api-access-bf9d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:47.156878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.156780    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9d6\" (UniqueName: \"kubernetes.io/projected/216f3637-5828-4d8f-845f-ae9a0f2170c8-kube-api-access-bf9d6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:47.339517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.294862    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca" pod "216f3637-5828-4d8f-845f-ae9a0f2170c8" (UID: "216f3637-5828-4d8f-845f-ae9a0f2170c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:47.339517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.310444    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f3637-5828-4d8f-845f-ae9a0f2170c8-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:47.339517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.339053    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.339517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.339464    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Oct 30 21:15:47.464410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.464278    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.477047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.476989    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.573745 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.571138    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.574044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.574011    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.708034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.707984    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:15:47.709307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.709283    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1e2bbf29-b9e4-4894-a351-b210f0b6b18e" containerName="revoke-keys"
Oct 30 21:15:47.709444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.709426    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2bbf29-b9e4-4894-a351-b210f0b6b18e" containerName="revoke-keys"
Oct 30 21:15:47.713507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.709636    7037 cpu_manager.go:414] "containerMap: removing container" podUID="216f3637-5828-4d8f-845f-ae9a0f2170c8" containerName="revoke-keys"
Oct 30 21:15:47.743582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.743506    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f3637-5828-4d8f-845f-ae9a0f2170c8" containerName="revoke-keys"
Oct 30 21:15:47.748508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.743966    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f3637-5828-4d8f-845f-ae9a0f2170c8" containerName="revoke-keys"
Oct 30 21:15:47.748699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.748682    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2bbf29-b9e4-4894-a351-b210f0b6b18e" containerName="revoke-keys"
Oct 30 21:15:47.854548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.716346    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.914448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.907746    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:15:47.914448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:47.907786    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:15:48.006265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.003457    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.009956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.005995    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.010314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.010291    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.010585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.010558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.010807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.010789    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.011025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.011005    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.011154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.011137    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.011278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.011262    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.011420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.011402    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpq4\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.011600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.011581    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.018476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.018444    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:48.029730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.029702    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:15:48.029919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.029904    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:15:48.030111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.030098    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:15:48.030225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.030214    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:15:48.030380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.030369    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:15:48.030586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.030572    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:15:48.031896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.031877    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.035407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.035380    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:15:48.037969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.037943    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:15:48.040695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.040677    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:15:48.062764 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:15:48.059606    7037 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f3637_5828_4d8f_845f_ae9a0f2170c8.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:15:48.068198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.068121    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:15:48.112891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.112853    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.113517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.113473    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cbq\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.113704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.113686    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.114616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.114594    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.115346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115327    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.115482 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115466    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.115659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115626    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.115769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115756    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpq4\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.115890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115877    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.115989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115977    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.116089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.116074    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.116213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.116201    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.116811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.116790    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.117061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.117046    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.118166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.118138    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.118595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.118572    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.118784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.118767    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.119649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.118372    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.119649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.115287    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.119649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.119144    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.119649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.119395    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.120354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.120325    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.120466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.120445    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.123915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.123892    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.125727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.125705    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.137081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.137059    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.151668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.151643    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpq4\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220386    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220517    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220556    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220601    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220883    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cbq\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220956    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.220994    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.221066    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.221123    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.221870    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.222189    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.222616    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.223515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.223467    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.224289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.223749    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.231959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.229221    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.239049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.238989    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.245970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.245945    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cbq\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.247285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.247244    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.411443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.411411    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:48.412248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.412217    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:48.783707 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:48.783669    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f14a956_ae09_485f_b5e0_8390677d1c92.slice/crio-5a03a406952a76c36cf9eded29811e65d7398315bc6250a8b8e92996d5ad783f WatchSource:0}: Error finding container 5a03a406952a76c36cf9eded29811e65d7398315bc6250a8b8e92996d5ad783f: Status 404 returned error can't find the container with id 5a03a406952a76c36cf9eded29811e65d7398315bc6250a8b8e92996d5ad783f
Oct 30 21:15:48.784281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.784249    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:15:48.786337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.786302    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.786450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.786372    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.786450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.786404    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.801874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.801840    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:15:48.804542 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:48.804512    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa94e31_814c_49d4_9cc8_0d5f6e11c620.slice/crio-60af31d05c2f0f964cded63a00632ccd101021674a9ef09edfe3b3b37a122fea WatchSource:0}: Error finding container 60af31d05c2f0f964cded63a00632ccd101021674a9ef09edfe3b3b37a122fea: Status 404 returned error can't find the container with id 60af31d05c2f0f964cded63a00632ccd101021674a9ef09edfe3b3b37a122fea
Oct 30 21:15:48.811166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.811126    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.811255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.811213    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.811300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.811256    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:15:48.974033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.972167    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"]
Oct 30 21:15:48.983953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:48.982682    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.001376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.001345    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:49.002160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.002136    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:15:49.014852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.014823    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"]
Oct 30 21:15:49.062600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.052233    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.062600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.052427    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxz6\" (UniqueName: \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.062600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.052578    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.092540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.090020    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" event={"ID":"0f14a956-ae09-485f-b5e0-8390677d1c92","Type":"ContainerStarted","Data":"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"}
Oct 30 21:15:49.092540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.090083    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" event={"ID":"0f14a956-ae09-485f-b5e0-8390677d1c92","Type":"ContainerStarted","Data":"5a03a406952a76c36cf9eded29811e65d7398315bc6250a8b8e92996d5ad783f"}
Oct 30 21:15:49.097253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.097217    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" event={"ID":"cfa94e31-814c-49d4-9cc8-0d5f6e11c620","Type":"ContainerStarted","Data":"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"}
Oct 30 21:15:49.097403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.097284    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" event={"ID":"cfa94e31-814c-49d4-9cc8-0d5f6e11c620","Type":"ContainerStarted","Data":"60af31d05c2f0f964cded63a00632ccd101021674a9ef09edfe3b3b37a122fea"}
Oct 30 21:15:49.152065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.151954    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podStartSLOduration=4.151935506 podStartE2EDuration="4.151935506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:49.149446086 +0000 UTC m=+892.560985525" watchObservedRunningTime="2026-10-30 21:15:49.151935506 +0000 UTC m=+892.563474946"
Oct 30 21:15:49.154025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.153993    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxz6\" (UniqueName: \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.154185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.154166    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.154901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.154878    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.156313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.155895    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.157939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.157912    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.190653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.190621    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxz6\" (UniqueName: \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\") pod \"maas-api-revoke-sub-d3a2332f-f467s\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.213470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.213421    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podStartSLOduration=4.213404632 podStartE2EDuration="4.213404632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:49.210877639 +0000 UTC m=+892.622417077" watchObservedRunningTime="2026-10-30 21:15:49.213404632 +0000 UTC m=+892.624944072"
Oct 30 21:15:49.339898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.339630    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:49.412041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.412010    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:49.412814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.412758    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:49.416553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.415653    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.416553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.415707    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:49.416553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.415984    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.416553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.416024    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:49.707457 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:49.707417    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode023f8b7_94dd_4f07_9baa_ab5c62ad33dd.slice/crio-69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9 WatchSource:0}: Error finding container 69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9: Status 404 returned error can't find the container with id 69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9
Oct 30 21:15:49.710801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.710775    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"]
Oct 30 21:15:49.885444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.885410    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"]
Oct 30 21:15:49.891388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.891368    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:49.985366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.984581    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"]
Oct 30 21:15:49.999190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.999155    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzgdq\" (UniqueName: \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:49.999608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:49.999562    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.000119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.000096    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.101764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.101724    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.101920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.101865    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzgdq\" (UniqueName: \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.101964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.101954    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.102508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.102468    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.104365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.104340    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.107945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.107921    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s" event={"ID":"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd","Type":"ContainerStarted","Data":"69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9"}
Oct 30 21:15:50.122372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.122349    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzgdq\" (UniqueName: \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\") pod \"maas-api-revoke-sub-6fc0c627-6ptzx\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.206753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.206723    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:50.419527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.413275    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:50.419527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.413333    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:50.450518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.422012    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:50.450518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.422070    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:50.781928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:50.781891    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"]
Oct 30 21:15:50.783999 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:15:50.783972    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080a587b_5ab2_4824_b347_289433a7fc29.slice/crio-c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302 WatchSource:0}: Error finding container c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302: Status 404 returned error can't find the container with id c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302
Oct 30 21:15:51.118563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.118519    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx" event={"ID":"080a587b-5ab2-4824-b347-289433a7fc29","Type":"ContainerStarted","Data":"c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302"}
Oct 30 21:15:51.120154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.120125    7037 generic.go:356] "Generic (PLEG): container finished" podID="e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" containerID="5e50aef54b12cafdfde0097007675e23619a255720bfaefbb2ab7cbdd4b4f463" exitCode=0
Oct 30 21:15:51.120295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.120186    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s" event={"ID":"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd","Type":"ContainerDied","Data":"5e50aef54b12cafdfde0097007675e23619a255720bfaefbb2ab7cbdd4b4f463"}
Oct 30 21:15:51.412318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.412241    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:51.412318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.412291    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:51.412833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.412807    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:51.412960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:51.412861    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:52.194804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.194167    7037 generic.go:356] "Generic (PLEG): container finished" podID="080a587b-5ab2-4824-b347-289433a7fc29" containerID="13f0e9ae2c5ce869f29a7607c70ca3d827b12353a8432ff0d1190de13336a98d" exitCode=0
Oct 30 21:15:52.195807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.194962    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx" event={"ID":"080a587b-5ab2-4824-b347-289433a7fc29","Type":"ContainerDied","Data":"13f0e9ae2c5ce869f29a7607c70ca3d827b12353a8432ff0d1190de13336a98d"}
Oct 30 21:15:52.416404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.416031    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:52.416404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.416113    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:52.416894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.416865    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:52.416969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.416945    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:52.579102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.578934    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:52.695362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.694991    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"]
Oct 30 21:15:52.696184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.696154    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" containerName="revoke-keys"
Oct 30 21:15:52.696184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.696184    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" containerName="revoke-keys"
Oct 30 21:15:52.696792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.696766    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" containerName="revoke-keys"
Oct 30 21:15:52.705871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.705840    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"]
Oct 30 21:15:52.706387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.706359    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.712675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.712646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\") pod \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") "
Oct 30 21:15:52.715263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.715239    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\") pod \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") "
Oct 30 21:15:52.715573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.715552    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\" (UniqueName: \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\") pod \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\" (UID: \"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd\") "
Oct 30 21:15:52.716611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.716272    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca" pod "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" (UID: "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:52.717388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.717366    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:52.717566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.717545    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca" pod "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" (UID: "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:52.721527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.721468    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6" pod "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" (UID: "e023f8b7-94dd-4f07-9baa-ab5c62ad33dd"). InnerVolumeSpecName "kube-api-access-czxz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:52.820730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.819331    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.822682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.822343    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.822682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.822511    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mvp\" (UniqueName: \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.822682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.822624    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czxz6\" (UniqueName: \"kubernetes.io/projected/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-kube-api-access-czxz6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:52.822682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.822635    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:52.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.924062    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.924247    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.924451    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mvp\" (UniqueName: \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.940516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.932299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.968038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.967926    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:52.971538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:52.971510    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mvp\" (UniqueName: \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\") pod \"maas-api-revoke-sub-70afd3c7-p9zqx\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:53.101405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.101368    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:53.212251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.211752    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"
Oct 30 21:15:53.212251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.212212    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s" event={"ID":"e023f8b7-94dd-4f07-9baa-ab5c62ad33dd","Type":"ContainerDied","Data":"69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9"}
Oct 30 21:15:53.213140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.212305    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a9d78cd540e83cc0118d45c2db800f03e7ab9d7eccb7758789e3a9a4cbe9e9"
Oct 30 21:15:53.412933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.412774    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:53.412933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.412820    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:53.412933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.412836    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:53.412933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.412865    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:53.498677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.496724    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"]
Oct 30 21:15:53.691144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.691094    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:53.804685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.804595    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\") pod \"080a587b-5ab2-4824-b347-289433a7fc29\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") "
Oct 30 21:15:53.804977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.804930    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\") pod \"080a587b-5ab2-4824-b347-289433a7fc29\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") "
Oct 30 21:15:53.805582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.805541    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\" (UniqueName: \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\") pod \"080a587b-5ab2-4824-b347-289433a7fc29\" (UID: \"080a587b-5ab2-4824-b347-289433a7fc29\") "
Oct 30 21:15:53.807093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.807068    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca" pod "080a587b-5ab2-4824-b347-289433a7fc29" (UID: "080a587b-5ab2-4824-b347-289433a7fc29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:53.808216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.808185    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:53.893309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.887058    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq" pod "080a587b-5ab2-4824-b347-289433a7fc29" (UID: "080a587b-5ab2-4824-b347-289433a7fc29"). InnerVolumeSpecName "kube-api-access-nzgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:53.911331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.911290    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzgdq\" (UniqueName: \"kubernetes.io/projected/080a587b-5ab2-4824-b347-289433a7fc29-kube-api-access-nzgdq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:53.918429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:53.918383    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca" pod "080a587b-5ab2-4824-b347-289433a7fc29" (UID: "080a587b-5ab2-4824-b347-289433a7fc29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:54.015845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.015805    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080a587b-5ab2-4824-b347-289433a7fc29-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:54.222832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.222799    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx" event={"ID":"65c9960a-5b82-44e1-aa8d-186f284d66f7","Type":"ContainerStarted","Data":"265f3fdb14de058a329bfb3767394d46063f7db6f6ff24f56af17edb12b20178"}
Oct 30 21:15:54.223186 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.222854    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx" event={"ID":"65c9960a-5b82-44e1-aa8d-186f284d66f7","Type":"ContainerStarted","Data":"0f76095b21bd2e639008c22430fe1d78f3e2235248dfadfec712de11e5723b46"}
Oct 30 21:15:54.225963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.225667    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx" event={"ID":"080a587b-5ab2-4824-b347-289433a7fc29","Type":"ContainerDied","Data":"c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302"}
Oct 30 21:15:54.225963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.225718    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6958388c4dbf0bfc0e52d50598e238fcdded8f8f379b6df70b252bc17a56302"
Oct 30 21:15:54.225963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.225777    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"
Oct 30 21:15:54.412554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.412311    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:54.412554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.412386    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:54.412888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.412864    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:54.413028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.413011    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:54.783408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:54.783336    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx" podStartSLOduration=2.7833139669999998 podStartE2EDuration="2.783313967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:54.280736131 +0000 UTC m=+897.692275574" watchObservedRunningTime="2026-10-30 21:15:54.783313967 +0000 UTC m=+898.194853407"
Oct 30 21:15:55.239679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.239638    7037 generic.go:356] "Generic (PLEG): container finished" podID="65c9960a-5b82-44e1-aa8d-186f284d66f7" containerID="265f3fdb14de058a329bfb3767394d46063f7db6f6ff24f56af17edb12b20178" exitCode=0
Oct 30 21:15:55.240108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.239733    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx" event={"ID":"65c9960a-5b82-44e1-aa8d-186f284d66f7","Type":"ContainerDied","Data":"265f3fdb14de058a329bfb3767394d46063f7db6f6ff24f56af17edb12b20178"}
Oct 30 21:15:55.412653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.412594    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:15:55.412813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.412649    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:15:55.412813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.412599    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:15:55.412813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:55.412739    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:15:56.519771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.519693    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:56.609926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.609811    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:56.610821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.610613    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:56.611775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.611748    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:15:56.614340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.614299    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\") pod \"65c9960a-5b82-44e1-aa8d-186f284d66f7\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") "
Oct 30 21:15:56.614453 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.614425    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\" (UniqueName: \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\") pod \"65c9960a-5b82-44e1-aa8d-186f284d66f7\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") "
Oct 30 21:15:56.614533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.614520    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\") pod \"65c9960a-5b82-44e1-aa8d-186f284d66f7\" (UID: \"65c9960a-5b82-44e1-aa8d-186f284d66f7\") "
Oct 30 21:15:56.615102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.615075    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca" pod "65c9960a-5b82-44e1-aa8d-186f284d66f7" (UID: "65c9960a-5b82-44e1-aa8d-186f284d66f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:56.616785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.616758    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca" pod "65c9960a-5b82-44e1-aa8d-186f284d66f7" (UID: "65c9960a-5b82-44e1-aa8d-186f284d66f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:56.616859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.616785    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp" pod "65c9960a-5b82-44e1-aa8d-186f284d66f7" (UID: "65c9960a-5b82-44e1-aa8d-186f284d66f7"). InnerVolumeSpecName "kube-api-access-65mvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:56.655564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.654867    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:56.655564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.655770    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:56.657203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.657178    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:15:56.716344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.716285    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:56.716344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.716315    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mvp\" (UniqueName: \"kubernetes.io/projected/65c9960a-5b82-44e1-aa8d-186f284d66f7-kube-api-access-65mvp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:56.716344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:56.716330    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9960a-5b82-44e1-aa8d-186f284d66f7-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:15:57.262564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:57.262441    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"
Oct 30 21:15:57.262564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:57.262461    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx" event={"ID":"65c9960a-5b82-44e1-aa8d-186f284d66f7","Type":"ContainerDied","Data":"0f76095b21bd2e639008c22430fe1d78f3e2235248dfadfec712de11e5723b46"}
Oct 30 21:15:57.262781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:15:57.262582    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f76095b21bd2e639008c22430fe1d78f3e2235248dfadfec712de11e5723b46"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.808556    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"]
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809708    7037 cpu_manager.go:414] "containerMap: removing container" podUID="65c9960a-5b82-44e1-aa8d-186f284d66f7" containerName="revoke-keys"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809730    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c9960a-5b82-44e1-aa8d-186f284d66f7" containerName="revoke-keys"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809775    7037 cpu_manager.go:414] "containerMap: removing container" podUID="080a587b-5ab2-4824-b347-289433a7fc29" containerName="revoke-keys"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809784    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080a587b-5ab2-4824-b347-289433a7fc29" containerName="revoke-keys"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809938    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c9960a-5b82-44e1-aa8d-186f284d66f7" containerName="revoke-keys"
Oct 30 21:16:03.813511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.809963    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="080a587b-5ab2-4824-b347-289433a7fc29" containerName="revoke-keys"
Oct 30 21:16:03.833980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.819217    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.888520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.877993    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srncx\" (UniqueName: \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.888520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.878404    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.888520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.878451    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.942518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.930423    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:16:03.985516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.981460    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.985516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.981541    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.985516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.981651    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srncx\" (UniqueName: \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:03.987705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:03.987059    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:04.053517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.008187    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:04.328514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.327822    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:04.352513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.347581    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.527515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.527045    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srncx\" (UniqueName: \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\") pod \"maas-api-revoke-sub-793863bc-pw5hc\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:04.554261 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:16:04.552216    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:16:04.554261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.552539    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"]
Oct 30 21:16:04.596335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.596223    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:04.628646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.628588    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.865582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.861436    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.937264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.937222    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.963753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:04.963711    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.271249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.271148    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.368881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.367735    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.369136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.369107    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.388060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.388020    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.402632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.402587    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.403258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.403234    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:16:05.685514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.684394    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.685514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.685120    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:16:05.972897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.948245    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.972897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.971242    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:16:05.976539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.974820    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.976539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.975172    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:16:06.004712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.979708    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:06.004712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:05.980038    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:16:06.376581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.318215    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:06.376581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.318599    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:16:06.411515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.392250    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:06.447667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.447635    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:16:06.459653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.459610    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:06.491904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.491874    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:16:06.529549 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.527826    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:06.529915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.529892    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:16:06.569141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.567972    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:06.692355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.692243    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:06.961832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.956064    7037 patch_prober.go:29] interesting pod/thanos-querier-7fd47999b4-r2bv9 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:06.961832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.956124    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-7fd47999b4-r2bv9" podUID="943e6093-44ff-4e3e-97d4-67a24ce560a0" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.14:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.044844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:06.969661    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.044844 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:06.992528    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e546b6_0de7_43b2_8a74_f98d50884b6f.slice/crio-c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121 WatchSource:0}: Error finding container c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121: Status 404 returned error can't find the container with id c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121
Oct 30 21:16:07.044844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.020645    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.044844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.022654    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.232071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.231974    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.370946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.370900    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.424854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.403587    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:07.477527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.472319    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:16:07.557629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.557561    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:07.584017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.583971    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:07.732673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.732624    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:07.987984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:07.987933    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:08.049678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.049616    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:08.050410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.050377    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:16:08.387519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.371662    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:08.387519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.373973    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:08.517646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.516764    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:08.527513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.526824    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:08.623654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.623605    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:08.699063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:08.698920    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:08.998575 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:16:08.987006    7037 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.268s"
Oct 30 21:16:09.220511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.213749    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc" event={"ID":"b3e546b6-0de7-43b2-8a74-f98d50884b6f","Type":"ContainerStarted","Data":"c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121"}
Oct 30 21:16:09.416675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.409772    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.465955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.465907    7037 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:09.466245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.466214    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:09.602483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.602445    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.603166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.603140    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.603713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.603674    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.604182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.604144    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:09.760341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.757009    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.760341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.758540    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.760341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.759621    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.792039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.788547    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:09.860388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.860355    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:09.882852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.882814    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc" event={"ID":"b3e546b6-0de7-43b2-8a74-f98d50884b6f","Type":"ContainerStarted","Data":"f094e89132ef908dbb7fba8862966ba19c13ac257def353391d1190f576bcfcc"}
Oct 30 21:16:09.883017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.882919    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:16:09.883017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.882953    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:16:09.883017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.882984    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883032    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883050    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"]
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883093    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883122    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883160    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:16:09.883180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.883180    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:16:09.890457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.890431    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:16:09.890638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.890546    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:09.892594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.892574    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:16:09.896734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.896710    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:16:09.907110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:09.907083    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-dd1e164d-gw-openshift-default-dockercfg-gjsxg\""
Oct 30 21:16:10.048088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.048047    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95cd\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.048623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.048591    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.048898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.048873    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.053082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.053036    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.053189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.053086    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.053256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.053211    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.053311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.053255    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.055098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.055071    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.055453 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.055429    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.165971    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166009    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166069    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166102    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166189    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166343    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166405    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166457    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95cd\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166557    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166604    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166732    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.166925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.166799    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.167103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.167082    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.167553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.167524    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.169915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.169884    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.170537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.170471    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.186984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.186943    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95cd\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.188985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.188956    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\") pod \"e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") " pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.228545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.210796    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:10.228545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.227128    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc" podStartSLOduration=7.227108619 podStartE2EDuration="7.227108619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:10.225007161 +0000 UTC m=+913.636546598" watchObservedRunningTime="2026-10-30 21:16:10.227108619 +0000 UTC m=+913.638648057"
Oct 30 21:16:10.583502 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:10.583453    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08d9d3a_72b9_4467_88a0_80cfd04496c9.slice/crio-0b7a979b49a3e32c28518cd35a5be21d58e1c790e2e26a909a443d89bd4b6885 WatchSource:0}: Error finding container 0b7a979b49a3e32c28518cd35a5be21d58e1c790e2e26a909a443d89bd4b6885: Status 404 returned error can't find the container with id 0b7a979b49a3e32c28518cd35a5be21d58e1c790e2e26a909a443d89bd4b6885
Oct 30 21:16:10.586830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.586796    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:10.586926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.586874    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:10.586967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.586926    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:10.587022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.586999    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:16:10.896398 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.896359    7037 generic.go:356] "Generic (PLEG): container finished" podID="b3e546b6-0de7-43b2-8a74-f98d50884b6f" containerID="f094e89132ef908dbb7fba8862966ba19c13ac257def353391d1190f576bcfcc" exitCode=0
Oct 30 21:16:10.896613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.896433    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc" event={"ID":"b3e546b6-0de7-43b2-8a74-f98d50884b6f","Type":"ContainerDied","Data":"f094e89132ef908dbb7fba8862966ba19c13ac257def353391d1190f576bcfcc"}
Oct 30 21:16:10.898555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.898530    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" event={"ID":"c08d9d3a-72b9-4467-88a0-80cfd04496c9","Type":"ContainerStarted","Data":"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"}
Oct 30 21:16:10.898666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.898571    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" event={"ID":"c08d9d3a-72b9-4467-88a0-80cfd04496c9","Type":"ContainerStarted","Data":"0b7a979b49a3e32c28518cd35a5be21d58e1c790e2e26a909a443d89bd4b6885"}
Oct 30 21:16:10.984046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:10.983998    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" podStartSLOduration=1.983983783 podStartE2EDuration="1.983983783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:10.978481574 +0000 UTC m=+914.390021011" watchObservedRunningTime="2026-10-30 21:16:10.983983783 +0000 UTC m=+914.395523220"
Oct 30 21:16:11.212091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:11.212000    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:11.218176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:11.218147    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:11.908878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:11.908847    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:11.909758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:11.909727    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:16:12.088168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.088137    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:12.208010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.207927    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\") pod \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") "
Oct 30 21:16:12.208010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.207995    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\" (UniqueName: \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\") pod \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") "
Oct 30 21:16:12.208276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.208257    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\") pod \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\" (UID: \"b3e546b6-0de7-43b2-8a74-f98d50884b6f\") "
Oct 30 21:16:12.208652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.208626    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca" pod "b3e546b6-0de7-43b2-8a74-f98d50884b6f" (UID: "b3e546b6-0de7-43b2-8a74-f98d50884b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:12.209000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.208982    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:12.210054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.210035    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca" pod "b3e546b6-0de7-43b2-8a74-f98d50884b6f" (UID: "b3e546b6-0de7-43b2-8a74-f98d50884b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:12.210203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.210183    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx" pod "b3e546b6-0de7-43b2-8a74-f98d50884b6f" (UID: "b3e546b6-0de7-43b2-8a74-f98d50884b6f"). InnerVolumeSpecName "kube-api-access-srncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:12.310604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.310576    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e546b6-0de7-43b2-8a74-f98d50884b6f-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:12.310604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.310604    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srncx\" (UniqueName: \"kubernetes.io/projected/b3e546b6-0de7-43b2-8a74-f98d50884b6f-kube-api-access-srncx\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:12.945559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.942600    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc" event={"ID":"b3e546b6-0de7-43b2-8a74-f98d50884b6f","Type":"ContainerDied","Data":"c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121"}
Oct 30 21:16:12.945559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.942660    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44cb5798646eb5804f5a5e99e9f03ed05a628e90dc1dd0e30420e7aa95a0121"
Oct 30 21:16:12.945559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:12.943406    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"
Oct 30 21:16:14.087621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.087582    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:16:14.088326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.088306    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b3e546b6-0de7-43b2-8a74-f98d50884b6f" containerName="revoke-keys"
Oct 30 21:16:14.088326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.088323    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e546b6-0de7-43b2-8a74-f98d50884b6f" containerName="revoke-keys"
Oct 30 21:16:14.088519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.088451    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e546b6-0de7-43b2-8a74-f98d50884b6f" containerName="revoke-keys"
Oct 30 21:16:14.092895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.092872    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.117225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.117190    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:16:14.122891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.122860    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dd1e164d-dockercfg-492ps\""
Oct 30 21:16:14.123097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.122966    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dd1e164d\""
Oct 30 21:16:14.151588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.151557    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.151790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.151714    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpnh\" (UniqueName: \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.277523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.267635    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpnh\" (UniqueName: \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.277523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.267878    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.277523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.268754    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.301700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.301663    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpnh\" (UniqueName: \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\") pod \"payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") " pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.414482 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.413352    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:14.538512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.535848    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"]
Oct 30 21:16:14.583637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.582732    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.604335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.598272    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:14.604335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.599158    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:16:14.639066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.628926    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"]
Oct 30 21:16:14.707591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.707517    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.707750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.707730    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shxp\" (UniqueName: \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.707795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.707766    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.809941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.809907    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.810396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.810375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shxp\" (UniqueName: \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.810499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.810411    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.811180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.811153    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.811606 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:14.811577    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad46b49_95ef_4dc4_860d_8af9ef564059.slice/crio-c4238f7f659116ed54165c1f48f2a94a4319eaa63eee9b48b7592f372ba4a36a WatchSource:0}: Error finding container c4238f7f659116ed54165c1f48f2a94a4319eaa63eee9b48b7592f372ba4a36a: Status 404 returned error can't find the container with id c4238f7f659116ed54165c1f48f2a94a4319eaa63eee9b48b7592f372ba4a36a
Oct 30 21:16:14.813413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.813393    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.814658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.814635    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:16:14.834226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.834201    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shxp\" (UniqueName: \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\") pod \"maas-api-revoke-sub-303f3994-lb7qx\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.936077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.935511    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:14.985058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.984963    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" event={"ID":"aad46b49-95ef-4dc4-860d-8af9ef564059","Type":"ContainerStarted","Data":"b972a5ac8e1bbfc302e316e95e1ac2b042c58824ecc29fb282d99c5c3d3eae0d"}
Oct 30 21:16:14.985058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.985031    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" event={"ID":"aad46b49-95ef-4dc4-860d-8af9ef564059","Type":"ContainerStarted","Data":"c4238f7f659116ed54165c1f48f2a94a4319eaa63eee9b48b7592f372ba4a36a"}
Oct 30 21:16:14.986191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:14.986166    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:15.032798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.032068    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" podStartSLOduration=1.032052874 podStartE2EDuration="1.032052874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:15.031298193 +0000 UTC m=+918.442837631" watchObservedRunningTime="2026-10-30 21:16:15.032052874 +0000 UTC m=+918.443592310"
Oct 30 21:16:15.270590 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:15.270541    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a318d2_0089_43bd_ade7_7d98e1fc68db.slice/crio-f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4 WatchSource:0}: Error finding container f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4: Status 404 returned error can't find the container with id f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4
Oct 30 21:16:15.275301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.275269    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"]
Oct 30 21:16:15.904459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.904424    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:16:15.909173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.909151    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:15.941090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.941059    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:16:15.995598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.995561    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx" event={"ID":"46a318d2-0089-43bd-ade7-7d98e1fc68db","Type":"ContainerStarted","Data":"c8f192d58481558bac46214cf6b9a4107160c1aad9e40da75546980fdaa5db10"}
Oct 30 21:16:15.995778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:15.995618    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx" event={"ID":"46a318d2-0089-43bd-ade7-7d98e1fc68db","Type":"ContainerStarted","Data":"f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4"}
Oct 30 21:16:16.041584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.041518    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx" podStartSLOduration=2.04148266 podStartE2EDuration="2.04148266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.040005147 +0000 UTC m=+919.451544598" watchObservedRunningTime="2026-10-30 21:16:16.04148266 +0000 UTC m=+919.453022098"
Oct 30 21:16:16.044349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044308    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044355    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69s92\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044514    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044677    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044748    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044779    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044852    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.044983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044933    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.045041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.044989    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.146720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.146680    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.146887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.146817    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.146944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.146930    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.146981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.146953    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69s92\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.147074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.147063    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.147151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.147140    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.147206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.147195    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.147250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.147219    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.147301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.147289    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.148041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.148021    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.148712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.148683    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.148834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.148717    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.148834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.148757    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.149128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.149105    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.150897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.150876    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.151014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.150992    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.171050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.170982    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.171177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.171118    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69s92\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.228730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.228692    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:16.544708 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:16.544664    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957e73ad_368b_4900_9839_ae9669b6858a.slice/crio-75384eda3204c993c26e6730cdceb206d39bff34b56b3b5a4fa18b735d85e6cd WatchSource:0}: Error finding container 75384eda3204c993c26e6730cdceb206d39bff34b56b3b5a4fa18b735d85e6cd: Status 404 returned error can't find the container with id 75384eda3204c993c26e6730cdceb206d39bff34b56b3b5a4fa18b735d85e6cd
Oct 30 21:16:16.547957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.547922    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:16.548035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.547999    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:16.548035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.548027    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:16.548321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:16.548298    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:16:17.021772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.021727    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" event={"ID":"957e73ad-368b-4900-9839-ae9669b6858a","Type":"ContainerStarted","Data":"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"}
Oct 30 21:16:17.021939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.021801    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" event={"ID":"957e73ad-368b-4900-9839-ae9669b6858a","Type":"ContainerStarted","Data":"75384eda3204c993c26e6730cdceb206d39bff34b56b3b5a4fa18b735d85e6cd"}
Oct 30 21:16:17.024664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.024638    7037 generic.go:356] "Generic (PLEG): container finished" podID="46a318d2-0089-43bd-ade7-7d98e1fc68db" containerID="c8f192d58481558bac46214cf6b9a4107160c1aad9e40da75546980fdaa5db10" exitCode=0
Oct 30 21:16:17.024811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.024722    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx" event={"ID":"46a318d2-0089-43bd-ade7-7d98e1fc68db","Type":"ContainerDied","Data":"c8f192d58481558bac46214cf6b9a4107160c1aad9e40da75546980fdaa5db10"}
Oct 30 21:16:17.069849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.069792    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podStartSLOduration=2.069774962 podStartE2EDuration="2.069774962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:17.065967626 +0000 UTC m=+920.477507065" watchObservedRunningTime="2026-10-30 21:16:17.069774962 +0000 UTC m=+920.481314402"
Oct 30 21:16:17.243515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.243275    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:17.243515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.243458    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:17.243790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:17.243537    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:18.316512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.295631    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:18.316512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.295687    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:18.521655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.472548    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:16:18.859535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.854593    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:18.891476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.890050    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-92d5d744\""
Oct 30 21:16:18.891476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.890214    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-92d5d744-dockercfg-rwjw5\""
Oct 30 21:16:18.922966 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.915680    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:16:18.955729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:18.955697    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:19.008738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.007614    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9df8\" (UniqueName: \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.008738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.008021    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.093034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.092955    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"
Oct 30 21:16:19.093034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.092995    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx" event={"ID":"46a318d2-0089-43bd-ade7-7d98e1fc68db","Type":"ContainerDied","Data":"f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4"}
Oct 30 21:16:19.093221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.093058    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f601fceaa28237161649710a449fbe6f1d6b82f041c1078761c9fde939870ec4"
Oct 30 21:16:19.109723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.109692    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\") pod \"46a318d2-0089-43bd-ade7-7d98e1fc68db\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") "
Oct 30 21:16:19.109864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.109832    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\" (UniqueName: \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\") pod \"46a318d2-0089-43bd-ade7-7d98e1fc68db\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") "
Oct 30 21:16:19.109950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.109913    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\") pod \"46a318d2-0089-43bd-ade7-7d98e1fc68db\" (UID: \"46a318d2-0089-43bd-ade7-7d98e1fc68db\") "
Oct 30 21:16:19.110404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.110378    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9df8\" (UniqueName: \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.110559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.110381    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca" pod "46a318d2-0089-43bd-ade7-7d98e1fc68db" (UID: "46a318d2-0089-43bd-ade7-7d98e1fc68db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:19.110559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.110554    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.110698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.110657    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:19.111182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.111159    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.111954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.111931    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca" pod "46a318d2-0089-43bd-ade7-7d98e1fc68db" (UID: "46a318d2-0089-43bd-ade7-7d98e1fc68db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:19.112099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.112081    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp" pod "46a318d2-0089-43bd-ade7-7d98e1fc68db" (UID: "46a318d2-0089-43bd-ade7-7d98e1fc68db"). InnerVolumeSpecName "kube-api-access-9shxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:19.154673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.154639    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9df8\" (UniqueName: \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\") pod \"payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.212343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.212301    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a318d2-0089-43bd-ade7-7d98e1fc68db-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:19.212343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.212334    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shxp\" (UniqueName: \"kubernetes.io/projected/46a318d2-0089-43bd-ade7-7d98e1fc68db-kube-api-access-9shxp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:19.254635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.247582    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:19.254635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.247663    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:19.254635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.248758    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:19.655848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:19.655820    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:16:20.103029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.102995    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" event={"ID":"36e37cbe-e571-4237-b688-5a3e6086aaed","Type":"ContainerStarted","Data":"1b2e13bbb9ffb959628c33acb72af5baf99874ae7cd48e34487981139f0f73d2"}
Oct 30 21:16:20.103218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.103044    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" event={"ID":"36e37cbe-e571-4237-b688-5a3e6086aaed","Type":"ContainerStarted","Data":"b19f7ea185df9d1057b9baaeab82afc707fdc2b3274b115145695e9a8ebd6614"}
Oct 30 21:16:20.103218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.103116    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:20.156304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.156237    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" podStartSLOduration=2.1562176 podStartE2EDuration="2.1562176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:20.149862348 +0000 UTC m=+923.561401786" watchObservedRunningTime="2026-10-30 21:16:20.1562176 +0000 UTC m=+923.567757038"
Oct 30 21:16:20.209143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.209109    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:16:20.209914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.209898    7037 cpu_manager.go:414] "containerMap: removing container" podUID="46a318d2-0089-43bd-ade7-7d98e1fc68db" containerName="revoke-keys"
Oct 30 21:16:20.209959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.209915    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a318d2-0089-43bd-ade7-7d98e1fc68db" containerName="revoke-keys"
Oct 30 21:16:20.210050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.210040    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a318d2-0089-43bd-ade7-7d98e1fc68db" containerName="revoke-keys"
Oct 30 21:16:20.214036 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.214013    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.263519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.259687    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4b62455c-dockercfg-sdkqk\""
Oct 30 21:16:20.263519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.259828    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:20.263519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.259863    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:16:20.263519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.259884    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:20.263519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.261508    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4b62455c\""
Oct 30 21:16:20.388573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.387969    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.388573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.388187    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw67q\" (UniqueName: \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.493718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.491652    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw67q\" (UniqueName: \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.493718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.492034    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.493718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.493178    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.513897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.513863    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw67q\" (UniqueName: \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\") pod \"payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.564764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.564734    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:20.956410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:20.956377    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:16:20.957289 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:20.957259    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed16483_21f2_478c_8e92_f7c2ff2fa7c9.slice/crio-ff14399b9f59772d6cb30fefd642f1c528df4953fb76d12cadb0b0bf5108b340 WatchSource:0}: Error finding container ff14399b9f59772d6cb30fefd642f1c528df4953fb76d12cadb0b0bf5108b340: Status 404 returned error can't find the container with id ff14399b9f59772d6cb30fefd642f1c528df4953fb76d12cadb0b0bf5108b340
Oct 30 21:16:21.115642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:21.115599    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" event={"ID":"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9","Type":"ContainerStarted","Data":"bcd6ba6c05b7f0a3de84cbeaf7772237959ca8d5675d347f290e9e0a1b572cf8"}
Oct 30 21:16:21.115831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:21.115670    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" event={"ID":"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9","Type":"ContainerStarted","Data":"ff14399b9f59772d6cb30fefd642f1c528df4953fb76d12cadb0b0bf5108b340"}
Oct 30 21:16:21.198773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:21.198718    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" podStartSLOduration=1.198700746 podStartE2EDuration="1.198700746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:21.193249377 +0000 UTC m=+924.604788814" watchObservedRunningTime="2026-10-30 21:16:21.198700746 +0000 UTC m=+924.610240182"
Oct 30 21:16:21.244575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:21.242004    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:21.244575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:21.242068    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:22.149515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:22.148679    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:22.229511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:22.229458    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:22.229801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:22.229770    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:23.229827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:23.229794    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:16:23.230247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:23.229847    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 21:16:24.233729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:24.233699    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:24.234171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:24.234152    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:24.234901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:24.234884    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:16:24.996935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:24.996894    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"]
Oct 30 21:16:25.002252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.002230    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.019000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.018971    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"]
Oct 30 21:16:25.025015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.024989    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:25.025126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.025052    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:16:25.091902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.091872    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcrq\" (UniqueName: \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.092055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.091910    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.092055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.092027    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.193558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.193486    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcrq\" (UniqueName: \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.193735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.193572    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.193735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.193657    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.194465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.194441    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.196037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.196008    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.212788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.212763    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcrq\" (UniqueName: \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\") pod \"maas-api-revoke-sub-31068716-c7ncp\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.318146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.318122    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:25.647849 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:25.647810    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c105dd7_4a76_4488_bc9f_5c0fddc65791.slice/crio-c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b WatchSource:0}: Error finding container c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b: Status 404 returned error can't find the container with id c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b
Oct 30 21:16:25.650192 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:25.650165    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"]
Oct 30 21:16:26.187607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:26.186107    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp" event={"ID":"9c105dd7-4a76-4488-bc9f-5c0fddc65791","Type":"ContainerStarted","Data":"bd4ec4b585b1d24349ce039d491cbd9e61fd1ecc52760c2583d1cff3ef256241"}
Oct 30 21:16:26.187607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:26.186188    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp" event={"ID":"9c105dd7-4a76-4488-bc9f-5c0fddc65791","Type":"ContainerStarted","Data":"c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b"}
Oct 30 21:16:26.228945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:26.228890    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp" podStartSLOduration=2.228877373 podStartE2EDuration="2.228877373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:26.223423554 +0000 UTC m=+929.634962990" watchObservedRunningTime="2026-10-30 21:16:26.228877373 +0000 UTC m=+929.640416844"
Oct 30 21:16:27.029244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.029211    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:16:27.196353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.196319    7037 generic.go:356] "Generic (PLEG): container finished" podID="9c105dd7-4a76-4488-bc9f-5c0fddc65791" containerID="bd4ec4b585b1d24349ce039d491cbd9e61fd1ecc52760c2583d1cff3ef256241" exitCode=0
Oct 30 21:16:27.196571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.196409    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp" event={"ID":"9c105dd7-4a76-4488-bc9f-5c0fddc65791","Type":"ContainerDied","Data":"bd4ec4b585b1d24349ce039d491cbd9e61fd1ecc52760c2583d1cff3ef256241"}
Oct 30 21:16:27.501938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.501903    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:16:27.506908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.506878    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.530474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.529899    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:16:27.532869 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.532373    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dd1e164d-serving-cert\""
Oct 30 21:16:27.635112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.635082    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.635281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.635265    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2s6\" (UniqueName: \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.635326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.635315    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.635460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.635436    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.739145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.737321    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2s6\" (UniqueName: \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.739145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.737409    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.739145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.737515    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.739145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.737686    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.744310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.744266    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.745172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.745145    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.752324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.752224    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.762990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.762964    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2s6\" (UniqueName: \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\") pod \"maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:27.842028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:27.841997    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:28.184341 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:16:28.184307    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30368a71_4282_4cbd_a4a1_1a3b812ab39f.slice/crio-550f0e9fd00bbed682712ae791ab1092cc46bb6b1dff8848805e3b25a0b53d5a WatchSource:0}: Error finding container 550f0e9fd00bbed682712ae791ab1092cc46bb6b1dff8848805e3b25a0b53d5a: Status 404 returned error can't find the container with id 550f0e9fd00bbed682712ae791ab1092cc46bb6b1dff8848805e3b25a0b53d5a
Oct 30 21:16:28.185998 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.185973    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:16:28.186219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.186206    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:16:28.206034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.206004    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" event={"ID":"30368a71-4282-4cbd-a4a1-1a3b812ab39f","Type":"ContainerStarted","Data":"550f0e9fd00bbed682712ae791ab1092cc46bb6b1dff8848805e3b25a0b53d5a"}
Oct 30 21:16:28.357801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.357705    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:28.558248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.558165    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\" (UniqueName: \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\") pod \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") "
Oct 30 21:16:28.558408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.558345    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\") pod \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") "
Oct 30 21:16:28.558610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.558592    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\") pod \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\" (UID: \"9c105dd7-4a76-4488-bc9f-5c0fddc65791\") "
Oct 30 21:16:28.559037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.559006    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca" pod "9c105dd7-4a76-4488-bc9f-5c0fddc65791" (UID: "9c105dd7-4a76-4488-bc9f-5c0fddc65791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:28.559375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.559355    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:28.560270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.560250    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca" pod "9c105dd7-4a76-4488-bc9f-5c0fddc65791" (UID: "9c105dd7-4a76-4488-bc9f-5c0fddc65791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:28.560368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.560352    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq" pod "9c105dd7-4a76-4488-bc9f-5c0fddc65791" (UID: "9c105dd7-4a76-4488-bc9f-5c0fddc65791"). InnerVolumeSpecName "kube-api-access-zhcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:28.660891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.660861    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c105dd7-4a76-4488-bc9f-5c0fddc65791-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:28.660891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:28.660888    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcrq\" (UniqueName: \"kubernetes.io/projected/9c105dd7-4a76-4488-bc9f-5c0fddc65791-kube-api-access-zhcrq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:16:29.217322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.217240    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"
Oct 30 21:16:29.217322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.217274    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp" event={"ID":"9c105dd7-4a76-4488-bc9f-5c0fddc65791","Type":"ContainerDied","Data":"c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b"}
Oct 30 21:16:29.217891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.217328    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9b38858bd4786011244a1ba0b5c3c1bd64f9d7e44c677ca405c82680783d45b"
Oct 30 21:16:29.219191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.219152    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" event={"ID":"30368a71-4282-4cbd-a4a1-1a3b812ab39f","Type":"ContainerStarted","Data":"07607d60842e802ee4dbfe1550da6f36b75e0dcb1985c106512c3d7da5eb8f2d"}
Oct 30 21:16:29.219719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.219629    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:29.267702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.267645    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" podStartSLOduration=1.591073915 podStartE2EDuration="2.267630492s" totalImagesPullingTime="676.556577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:29.264103274 +0000 UTC m=+932.675642711" watchObservedRunningTime="2026-10-30 21:16:29.267630492 +0000 UTC m=+932.679169937"
Oct 30 21:16:29.293613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.293573    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:16:29.294387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.294368    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9c105dd7-4a76-4488-bc9f-5c0fddc65791" containerName="revoke-keys"
Oct 30 21:16:29.294387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.294387    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c105dd7-4a76-4488-bc9f-5c0fddc65791" containerName="revoke-keys"
Oct 30 21:16:29.294554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.294539    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c105dd7-4a76-4488-bc9f-5c0fddc65791" containerName="revoke-keys"
Oct 30 21:16:29.298602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.298584    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.318235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.318206    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:16:29.318470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.318446    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:16:29.318644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.318469    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:16:29.318644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.318535    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wmcm5\""
Oct 30 21:16:29.320944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.320918    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:16:29.476097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.475992    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476070    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476206    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476228    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476248    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476349    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8zj\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.476427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.476371    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.577990    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578026    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578045    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578173    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8zj\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578199    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578272    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.578413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.578347    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.579138 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.579109    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.580871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.580740    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.580871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.580827    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.581040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.580997    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.581079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.581050    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.597612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.597589    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.597773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.597751    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8zj\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dtvjx\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.612679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.612656    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:29.940622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:29.940477    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:16:30.229515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:30.229405    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" event={"ID":"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56","Type":"ContainerStarted","Data":"342c7065d19ff800d062375f75e960cfe7909f36b6fc8582426085fe36a43005"}
Oct 30 21:16:30.860336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:30.860303    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:16:30.865919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:30.865888    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:30.897948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:30.897919    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:16:31.003861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.003822    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004063    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004127    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004214    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004273    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004443    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6cl\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004608    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004644    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.004732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.004690    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106013    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106077    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106113    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106161    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106264    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6cl\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106398 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106362    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106394    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106475 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106435    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.106574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.106519    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.107417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.107388    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.107755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.107730    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.107860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.107754    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.107860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.107777    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.108018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.107971    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.109370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.109345    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.112031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.111983    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.119109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.119091    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:16:31.137933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.137912    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6cl\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.142346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.142321    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.194779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.194740    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:31.529124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.528964    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:31.529124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.528992    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:16:31.529124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.529070    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:31.529562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:31.529138    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:32.255099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:32.255064    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" event={"ID":"04fd1d12-f119-4cce-9e58-53222ca65c74","Type":"ContainerStarted","Data":"37f5306ebf1c2e62e0dadf930aef20480efd4d0cb8def7490759febed7c2cf5e"}
Oct 30 21:16:32.255276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:32.255127    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" event={"ID":"04fd1d12-f119-4cce-9e58-53222ca65c74","Type":"ContainerStarted","Data":"c8f6ae7f9dd987b50436387a92fc71dd178eb7f4b667d5a0053f2f1bd8ec5d62"}
Oct 30 21:16:32.331777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:32.331724    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podStartSLOduration=2.331706483 podStartE2EDuration="2.331706483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.317465003 +0000 UTC m=+935.729004443" watchObservedRunningTime="2026-10-30 21:16:32.331706483 +0000 UTC m=+935.743245921"
Oct 30 21:16:33.164176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.164143    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:16:33.197866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.195743    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:33.198162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.198139    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:33.198217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.198189    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:33.586162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.586117    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:33.586272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:33.586210    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:16:33.696369 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:16:33.696337    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:16:34.196108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.196065    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:34.196790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.196128    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:34.290954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.290920    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" event={"ID":"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56","Type":"ContainerStarted","Data":"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"}
Oct 30 21:16:34.291429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.291407    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:34.293246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.293220    7037 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dtvjx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:16:34.293353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.293263    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:16:34.455870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:34.455754    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" podStartSLOduration=1.8110595059999999 podStartE2EDuration="5.455733022s" totalImagesPullingTime="3.644673516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:34.399879624 +0000 UTC m=+937.811419061" watchObservedRunningTime="2026-10-30 21:16:34.455733022 +0000 UTC m=+937.867272459"
Oct 30 21:16:35.195596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:35.195559    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:35.195753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:35.195624    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:35.257859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:35.257831    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:16:35.305001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:35.304970    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:16:36.198377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:36.198337    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:36.198694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:36.198663    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:37.195286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:37.195246    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:37.195803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:37.195312    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:38.195558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:38.195522    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:16:38.195946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:38.195577    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 30 21:16:39.659510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:39.659230    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:39.664351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:39.664325    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:39.669765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:39.669732    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:16:40.355741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.355685    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:40.363521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.356031    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:40.575669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.575612    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:40.612793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.612689    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:40.776866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.774581    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:40.776866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:40.776605    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.040820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.040718    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.162918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.162867    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:41.486545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.463598    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:41.486545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.464539    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.557443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.555843    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.603505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.603443    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.631522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.626820    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.650529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.642561    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:16:41.832554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.832504    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:41.833198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.833158    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:16:41.953513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:41.915908    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:16:42.000724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.000683    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:16:42.469539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.468454    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:16:42.495578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.270580    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:42.754648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.440211    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:16:42.834534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.516674    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:42.916906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.916864    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 30 21:16:42.917163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.624967    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:42.917659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.649214    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:43.000823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.000786    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:16:43.001156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.339664    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:43.021605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.680821    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:43.055517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.044974    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:43.055517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.045811    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:16:43.104013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.103979    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:16:43.125833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.125799    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:16:43.164244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.164205    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:16:43.282574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:42.624889    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 21:16:43.412983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.412934    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 21:16:43.516532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.496704    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:16:43.658716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.658666    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:16:43.742013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.741962    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:43.835650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.835581    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:43.917778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:43.917567    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.018911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.001695    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.295571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.056568    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.295571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.056607    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.295571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.056625    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.295571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.166111    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.638512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.635071    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.754517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.753786    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.918574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.881671    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.948517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:44.945143    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.297696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.297563    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.310798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.310756    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.313605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.313563    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.323832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.323794    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.324057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.324023    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.747524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.637503    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.981498 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.777287    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.988569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.973089    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.001600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:45.973144    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.327576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.315661    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.327576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.316172    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.377085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.347797    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.377085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.350768    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.417758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.417692    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.985401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:46.985351    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.029830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.029781    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.063516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.033931    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.104697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.104643    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.105142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.105118    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.325513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.321427    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.335171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.325051    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.350516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.348693    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.358050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.351580    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.472528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.449793    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.986123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:47.986074    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.056743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.046583    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.084525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.082762    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.127515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.105567    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.326307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.326261    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.328640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.328593    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.338083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.338045    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.389335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.354241    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.464945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.354581    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.481515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.460591    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.964511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:48.960792    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:16:49.023339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.019687    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:16:49.027165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.026908    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.031517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.029939    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:16:49.034533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.034426    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:16:49.048507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.045194    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:16:49.048507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.048459    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:16:49.062641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.062620    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:16:49.063277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.063239    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.094907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.094880    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:16:49.095655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:16:49.095631    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:17:01.036836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.036801    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"]
Oct 30 21:17:01.058774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.058707    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a25340-rb242"]
Oct 30 21:17:01.063714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.063659    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"]
Oct 30 21:17:01.070172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.070148    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.071987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.071961    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"]
Oct 30 21:17:01.084866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.084844    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:01.085109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.085092    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:17:01.206356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.206322    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.206571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.206397    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ws6w\" (UniqueName: \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.206571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.206476    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.308184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.308102    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.308184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.308166    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ws6w\" (UniqueName: \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.308418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.308201    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.308909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.308886    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.310596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.310564    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.327999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.327976    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ws6w\" (UniqueName: \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\") pod \"maas-api-revoke-sub-91666752-99cvk\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.386056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.386027    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:01.697683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:01.697641    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"]
Oct 30 21:17:02.220185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:02.220104    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk" event={"ID":"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4","Type":"ContainerStarted","Data":"4a4877a09e5e3c48d187a242b7ccc01f6d5a02ef87800778fbb38727e538b047"}
Oct 30 21:17:02.220185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:02.220158    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk" event={"ID":"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4","Type":"ContainerStarted","Data":"8e7378ae0de33983785d9fc1e24300d2d8340fb7907342a10301898198db5213"}
Oct 30 21:17:02.266464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:02.266410    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk" podStartSLOduration=1.266395755 podStartE2EDuration="1.266395755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:02.260852749 +0000 UTC m=+965.672392183" watchObservedRunningTime="2026-10-30 21:17:02.266395755 +0000 UTC m=+965.677935226"
Oct 30 21:17:02.710163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:02.710131    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce075a04-45a2-4af0-ba64-2dddd9f9d962" path="/var/lib/kubelet/pods/ce075a04-45a2-4af0-ba64-2dddd9f9d962/volumes"
Oct 30 21:17:03.233260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:03.233228    7037 generic.go:356] "Generic (PLEG): container finished" podID="700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" containerID="4a4877a09e5e3c48d187a242b7ccc01f6d5a02ef87800778fbb38727e538b047" exitCode=0
Oct 30 21:17:03.233708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:03.233309    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk" event={"ID":"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4","Type":"ContainerDied","Data":"4a4877a09e5e3c48d187a242b7ccc01f6d5a02ef87800778fbb38727e538b047"}
Oct 30 21:17:03.673369 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:17:03.673329    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:17:04.435893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.435867    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:04.464794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.464766    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\") pod \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") "
Oct 30 21:17:04.464947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.464869    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\" (UniqueName: \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\") pod \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") "
Oct 30 21:17:04.465037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.465020    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\") pod \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\" (UID: \"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4\") "
Oct 30 21:17:04.465189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.465163    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca" pod "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" (UID: "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:04.465808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.465784    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:04.467322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.467297    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca" pod "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" (UID: "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:04.468007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.467975    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w" pod "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" (UID: "700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4"). InnerVolumeSpecName "kube-api-access-9ws6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:04.567425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.567395    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ws6w\" (UniqueName: \"kubernetes.io/projected/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-kube-api-access-9ws6w\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:04.567570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:04.567428    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.265713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:05.265682    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"
Oct 30 21:17:05.265713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:05.265703    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk" event={"ID":"700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4","Type":"ContainerDied","Data":"8e7378ae0de33983785d9fc1e24300d2d8340fb7907342a10301898198db5213"}
Oct 30 21:17:05.265932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:05.265751    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7378ae0de33983785d9fc1e24300d2d8340fb7907342a10301898198db5213"
Oct 30 21:17:08.210833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.210802    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"]
Oct 30 21:17:08.211827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.211798    7037 cpu_manager.go:414] "containerMap: removing container" podUID="700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" containerName="revoke-keys"
Oct 30 21:17:08.211827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.211820    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" containerName="revoke-keys"
Oct 30 21:17:08.212001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.211976    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" containerName="revoke-keys"
Oct 30 21:17:08.216412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.216393    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.233535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.233322    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"]
Oct 30 21:17:08.239529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.237803    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:08.239529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.237871    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:17:08.340752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.340719    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcq4m\" (UniqueName: \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.340916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.340770    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.341046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.341028    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.442942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.442909    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.443102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.443033    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq4m\" (UniqueName: \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.443102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.443067    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.443723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.443702    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.445333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.445309    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.462959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.462887    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq4m\" (UniqueName: \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\") pod \"maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.536755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.536727    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:08.837112 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:17:08.837066    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331f13a5_3319_44f1_8d43_298ff083e4f7.slice/crio-c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed WatchSource:0}: Error finding container c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed: Status 404 returned error can't find the container with id c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed
Oct 30 21:17:08.842273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:08.842244    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"]
Oct 30 21:17:09.308555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:09.308519    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l" event={"ID":"331f13a5-3319-44f1-8d43-298ff083e4f7","Type":"ContainerStarted","Data":"bc724c043029332435b9d6ffad7eae20eec6acfc9a08928cacfd965c40ee8db6"}
Oct 30 21:17:09.309010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:09.308582    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l" event={"ID":"331f13a5-3319-44f1-8d43-298ff083e4f7","Type":"ContainerStarted","Data":"c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed"}
Oct 30 21:17:09.348821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:09.348766    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l" podStartSLOduration=1.348749878 podStartE2EDuration="1.348749878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:09.34535283 +0000 UTC m=+972.756892268" watchObservedRunningTime="2026-10-30 21:17:09.348749878 +0000 UTC m=+972.760289319"
Oct 30 21:17:10.321853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:10.321816    7037 generic.go:356] "Generic (PLEG): container finished" podID="331f13a5-3319-44f1-8d43-298ff083e4f7" containerID="bc724c043029332435b9d6ffad7eae20eec6acfc9a08928cacfd965c40ee8db6" exitCode=0
Oct 30 21:17:10.322279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:10.321900    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l" event={"ID":"331f13a5-3319-44f1-8d43-298ff083e4f7","Type":"ContainerDied","Data":"bc724c043029332435b9d6ffad7eae20eec6acfc9a08928cacfd965c40ee8db6"}
Oct 30 21:17:11.542273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.542244    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:11.713404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.713330    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\") pod \"331f13a5-3319-44f1-8d43-298ff083e4f7\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") "
Oct 30 21:17:11.713559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.713520    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\" (UniqueName: \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\") pod \"331f13a5-3319-44f1-8d43-298ff083e4f7\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") "
Oct 30 21:17:11.713637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.713623    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\") pod \"331f13a5-3319-44f1-8d43-298ff083e4f7\" (UID: \"331f13a5-3319-44f1-8d43-298ff083e4f7\") "
Oct 30 21:17:11.713974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.713948    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca" pod "331f13a5-3319-44f1-8d43-298ff083e4f7" (UID: "331f13a5-3319-44f1-8d43-298ff083e4f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:11.714301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.714283    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:11.715481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.715454    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca" pod "331f13a5-3319-44f1-8d43-298ff083e4f7" (UID: "331f13a5-3319-44f1-8d43-298ff083e4f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:11.715602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.715517    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m" pod "331f13a5-3319-44f1-8d43-298ff083e4f7" (UID: "331f13a5-3319-44f1-8d43-298ff083e4f7"). InnerVolumeSpecName "kube-api-access-wcq4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:11.815422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.815388    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcq4m\" (UniqueName: \"kubernetes.io/projected/331f13a5-3319-44f1-8d43-298ff083e4f7-kube-api-access-wcq4m\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:11.815422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:11.815422    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/331f13a5-3319-44f1-8d43-298ff083e4f7-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:12.032717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.032633    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"]
Oct 30 21:17:12.038198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.038170    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-286b0cd4-mkgsh"]
Oct 30 21:17:12.342103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.342066    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"
Oct 30 21:17:12.342103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.342093    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l" event={"ID":"331f13a5-3319-44f1-8d43-298ff083e4f7","Type":"ContainerDied","Data":"c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed"}
Oct 30 21:17:12.342320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.342142    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92d9e926eb8228c75a35915d2beb63eac2a498597cb73ce0ec3892dbf7b60ed"
Oct 30 21:17:12.714219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:12.714134    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0813c647-e867-49dc-96af-4df538f358a2" path="/var/lib/kubelet/pods/0813c647-e867-49dc-96af-4df538f358a2/volumes"
Oct 30 21:17:14.208825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:14.208785    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:17:15.066595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:15.066564    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"]
Oct 30 21:17:15.071153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:15.071120    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940f171f-4wk5t"]
Oct 30 21:17:16.043187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:16.043156    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"]
Oct 30 21:17:16.052977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:16.052943    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4460b9c1-t5vq8"]
Oct 30 21:17:16.778517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:16.776990    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d176d15-1437-436d-8cc3-f3bbd3277108" path="/var/lib/kubelet/pods/3d176d15-1437-436d-8cc3-f3bbd3277108/volumes"
Oct 30 21:17:16.778517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:16.777755    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ca0c5a-cea7-40a4-99be-cbc904c3e860" path="/var/lib/kubelet/pods/a7ca0c5a-cea7-40a4-99be-cbc904c3e860/volumes"
Oct 30 21:17:17.862515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:17.860939    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:17:17.923637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:17.922588    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.089518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.048237    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.277183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.276965    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 30 21:17:18.292529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.292467    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.297154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.296725    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.302402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.301579    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.449839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.449221    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.647253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.646592    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.763972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.762769    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.938604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.866640    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:18.975534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:18.951853    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:19.059524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.051818    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:19.239425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.239194    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" podUID="aad46b49-95ef-4dc4-860d-8af9ef564059" containerName="payload-processing" containerID="cri-o://b972a5ac8e1bbfc302e316e95e1ac2b042c58824ecc29fb282d99c5c3d3eae0d" gracePeriod=25
Oct 30 21:17:19.383895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.370476    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:17:19.430903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.403148    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:19.439163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.432195    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523582    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523690    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523732    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523752    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"]
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523796    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523833    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:17:19.526580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.523870    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:17:19.571516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.540679    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.608721    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.625445    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f61f33e-dn52g"]
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.047470    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.690357    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.752635    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.884361    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.774123    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.048805    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.048855    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:19.990575    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.059538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.046587    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 30 21:17:20.072531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.068600    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:17:20.387569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.387314    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.453544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.436648    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:20.482074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.450610    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:17:20.482657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:20.454348    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:17:21.168949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.168842    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:17:21.223518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.222547    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:17:21.223518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.222580    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:17:21.249525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.232744    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:17:21.249525 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.222547    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:21.400219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.400183    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28888c2d-0013-4038-8fb9-c1f77eb7bf2f" path="/var/lib/kubelet/pods/28888c2d-0013-4038-8fb9-c1f77eb7bf2f/volumes"
Oct 30 21:17:21.409550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.409513    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:21.481258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.481126    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:21.482550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.482516    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:21.492534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.492465    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:17:21.816885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.804379    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"]
Oct 30 21:17:21.816885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:21.813561    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.213519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.179187    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:17:22.272633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.215654    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10ee2a-8qdz9"]
Oct 30 21:17:22.272921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.223182    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:17:22.275851 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.223420    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.291513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.237937    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.291513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.249754    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.493400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.491557    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.493400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.492027    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:17:22.493400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.492600    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.495806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.494853    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:17:22.521748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.499557    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.521748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.509335    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:22.547590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.528544    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:17:22.547590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.546107    7037 generic.go:356] "Generic (PLEG): container finished" podID="aad46b49-95ef-4dc4-860d-8af9ef564059" containerID="b972a5ac8e1bbfc302e316e95e1ac2b042c58824ecc29fb282d99c5c3d3eae0d" exitCode=0
Oct 30 21:17:22.547590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.547422    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" event={"ID":"aad46b49-95ef-4dc4-860d-8af9ef564059","Type":"ContainerDied","Data":"b972a5ac8e1bbfc302e316e95e1ac2b042c58824ecc29fb282d99c5c3d3eae0d"}
Oct 30 21:17:22.613906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.613605    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 30 21:17:22.786983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.785648    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Oct 30 21:17:22.955609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.951279    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 30 21:17:23.020180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.988412    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23174b83-a192-4abb-aeee-f46bed977cac" path="/var/lib/kubelet/pods/23174b83-a192-4abb-aeee-f46bed977cac/volumes"
Oct 30 21:17:23.020180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.989120    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:17:23.020180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.989197    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:17:23.020180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.989244    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:17:23.020180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:22.989290    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:17:23.203554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.189760    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:17:23.216567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.216537    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:17:23.217970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.217115    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:17:23.217970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.217162    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:17:23.245520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.235139    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:17:23.245520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.235413    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" podUID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" containerName="maas-api" containerID="cri-o://07607d60842e802ee4dbfe1550da6f36b75e0dcb1985c106512c3d7da5eb8f2d" gracePeriod=30
Oct 30 21:17:23.294122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.290808    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:17:23.399171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.399141    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:17:23.399640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.399614    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerName="istio-proxy" containerID="cri-o://1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd" gracePeriod=30
Oct 30 21:17:23.455565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.452203    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:17:23.574625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.574572    7037 generic.go:356] "Generic (PLEG): container finished" podID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" containerID="07607d60842e802ee4dbfe1550da6f36b75e0dcb1985c106512c3d7da5eb8f2d" exitCode=0
Oct 30 21:17:23.574955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.574927    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" event={"ID":"30368a71-4282-4cbd-a4a1-1a3b812ab39f","Type":"ContainerDied","Data":"07607d60842e802ee4dbfe1550da6f36b75e0dcb1985c106512c3d7da5eb8f2d"}
Oct 30 21:17:23.578860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.577575    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj" event={"ID":"aad46b49-95ef-4dc4-860d-8af9ef564059","Type":"ContainerDied","Data":"c4238f7f659116ed54165c1f48f2a94a4319eaa63eee9b48b7592f372ba4a36a"}
Oct 30 21:17:23.578860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.577631    7037 scope.go:122] "RemoveContainer" containerID="b972a5ac8e1bbfc302e316e95e1ac2b042c58824ecc29fb282d99c5c3d3eae0d"
Oct 30 21:17:23.578860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.577647    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"
Oct 30 21:17:23.578860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.578312    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\" (UniqueName: \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\") pod \"aad46b49-95ef-4dc4-860d-8af9ef564059\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") "
Oct 30 21:17:23.578860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.578382    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\") pod \"aad46b49-95ef-4dc4-860d-8af9ef564059\" (UID: \"aad46b49-95ef-4dc4-860d-8af9ef564059\") "
Oct 30 21:17:23.579917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.579821    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume" pod "aad46b49-95ef-4dc4-860d-8af9ef564059" (UID: "aad46b49-95ef-4dc4-860d-8af9ef564059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:23.586684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.586649    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh" pod "aad46b49-95ef-4dc4-860d-8af9ef564059" (UID: "aad46b49-95ef-4dc4-860d-8af9ef564059"). InnerVolumeSpecName "kube-api-access-cnpnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:23.681695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.681663    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpnh\" (UniqueName: \"kubernetes.io/projected/aad46b49-95ef-4dc4-860d-8af9ef564059-kube-api-access-cnpnh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:23.681695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:23.681694    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad46b49-95ef-4dc4-860d-8af9ef564059-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:24.080513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.076681    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:17:24.105635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.105054    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:17:24.116546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.116340    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dd1e164d-5d78f9cfd-h7dcj"]
Oct 30 21:17:24.180321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.180281    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\") pod \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") "
Oct 30 21:17:24.180538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.180420    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\") pod \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") "
Oct 30 21:17:24.180636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.180533    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\" (UniqueName: \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\") pod \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") "
Oct 30 21:17:24.180808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.180783    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\") pod \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\" (UID: \"30368a71-4282-4cbd-a4a1-1a3b812ab39f\") "
Oct 30 21:17:24.182012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.181984    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca" pod "30368a71-4282-4cbd-a4a1-1a3b812ab39f" (UID: "30368a71-4282-4cbd-a4a1-1a3b812ab39f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:24.184907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.184865    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6" pod "30368a71-4282-4cbd-a4a1-1a3b812ab39f" (UID: "30368a71-4282-4cbd-a4a1-1a3b812ab39f"). InnerVolumeSpecName "kube-api-access-4b2s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:24.185435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.185401    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs" pod "30368a71-4282-4cbd-a4a1-1a3b812ab39f" (UID: "30368a71-4282-4cbd-a4a1-1a3b812ab39f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:24.187386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.187350    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls" pod "30368a71-4282-4cbd-a4a1-1a3b812ab39f" (UID: "30368a71-4282-4cbd-a4a1-1a3b812ab39f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:24.284426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.284318    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:24.284426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.284349    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30368a71-4282-4cbd-a4a1-1a3b812ab39f-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:24.284426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.284363    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b2s6\" (UniqueName: \"kubernetes.io/projected/30368a71-4282-4cbd-a4a1-1a3b812ab39f-kube-api-access-4b2s6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:24.284426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.284387    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30368a71-4282-4cbd-a4a1-1a3b812ab39f-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:24.600959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.600917    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw" event={"ID":"30368a71-4282-4cbd-a4a1-1a3b812ab39f","Type":"ContainerDied","Data":"550f0e9fd00bbed682712ae791ab1092cc46bb6b1dff8848805e3b25a0b53d5a"}
Oct 30 21:17:24.601296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.601030    7037 scope.go:122] "RemoveContainer" containerID="07607d60842e802ee4dbfe1550da6f36b75e0dcb1985c106512c3d7da5eb8f2d"
Oct 30 21:17:24.601296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.601177    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"
Oct 30 21:17:24.781020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.780986    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad46b49-95ef-4dc4-860d-8af9ef564059" path="/var/lib/kubelet/pods/aad46b49-95ef-4dc4-860d-8af9ef564059/volumes"
Oct 30 21:17:24.782282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.782256    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:17:24.782478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:24.782464    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd1e164d-68d59c7dd7-vpjpw"]
Oct 30 21:17:26.712157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:26.712125    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" path="/var/lib/kubelet/pods/30368a71-4282-4cbd-a4a1-1a3b812ab39f/volumes"
Oct 30 21:17:27.529072 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.528994    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" podUID="36e37cbe-e571-4237-b688-5a3e6086aaed" containerName="payload-pre-processing" containerID="cri-o://1b2e13bbb9ffb959628c33acb72af5baf99874ae7cd48e34487981139f0f73d2" gracePeriod=25
Oct 30 21:17:27.657319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.657282    7037 generic.go:356] "Generic (PLEG): container finished" podID="36e37cbe-e571-4237-b688-5a3e6086aaed" containerID="1b2e13bbb9ffb959628c33acb72af5baf99874ae7cd48e34487981139f0f73d2" exitCode=0
Oct 30 21:17:27.657479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.657459    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" event={"ID":"36e37cbe-e571-4237-b688-5a3e6086aaed","Type":"ContainerDied","Data":"1b2e13bbb9ffb959628c33acb72af5baf99874ae7cd48e34487981139f0f73d2"}
Oct 30 21:17:27.818821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.818778    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"]
Oct 30 21:17:27.835927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.835886    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92d5d744-l6r5l"]
Oct 30 21:17:27.873587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.873541    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:17:27.950693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.950648    7037 patch_prober.go:29] interesting pod/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:27.950850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.950728    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:27.963989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.963953    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\") pod \"36e37cbe-e571-4237-b688-5a3e6086aaed\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") "
Oct 30 21:17:27.964155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.964006    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\" (UniqueName: \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\") pod \"36e37cbe-e571-4237-b688-5a3e6086aaed\" (UID: \"36e37cbe-e571-4237-b688-5a3e6086aaed\") "
Oct 30 21:17:27.964445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.964420    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume" pod "36e37cbe-e571-4237-b688-5a3e6086aaed" (UID: "36e37cbe-e571-4237-b688-5a3e6086aaed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:27.964896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.964865    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36e37cbe-e571-4237-b688-5a3e6086aaed-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:27.970918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:27.970889    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8" pod "36e37cbe-e571-4237-b688-5a3e6086aaed" (UID: "36e37cbe-e571-4237-b688-5a3e6086aaed"). InnerVolumeSpecName "kube-api-access-x9df8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:28.066790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.066753    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9df8\" (UniqueName: \"kubernetes.io/projected/36e37cbe-e571-4237-b688-5a3e6086aaed-kube-api-access-x9df8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:28.291664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.291539    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" podUID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" containerName="payload-pre-processing" containerID="cri-o://bcd6ba6c05b7f0a3de84cbeaf7772237959ca8d5675d347f290e9e0a1b572cf8" gracePeriod=25
Oct 30 21:17:28.775247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.774504    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331f13a5-3319-44f1-8d43-298ff083e4f7" path="/var/lib/kubelet/pods/331f13a5-3319-44f1-8d43-298ff083e4f7/volumes"
Oct 30 21:17:28.775247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.775203    7037 generic.go:356] "Generic (PLEG): container finished" podID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" containerID="bcd6ba6c05b7f0a3de84cbeaf7772237959ca8d5675d347f290e9e0a1b572cf8" exitCode=0
Oct 30 21:17:28.775537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.775397    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" event={"ID":"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9","Type":"ContainerDied","Data":"bcd6ba6c05b7f0a3de84cbeaf7772237959ca8d5675d347f290e9e0a1b572cf8"}
Oct 30 21:17:28.777798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.777746    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr" event={"ID":"36e37cbe-e571-4237-b688-5a3e6086aaed","Type":"ContainerDied","Data":"b19f7ea185df9d1057b9baaeab82afc707fdc2b3274b115145695e9a8ebd6614"}
Oct 30 21:17:28.777941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.777812    7037 scope.go:122] "RemoveContainer" containerID="1b2e13bbb9ffb959628c33acb72af5baf99874ae7cd48e34487981139f0f73d2"
Oct 30 21:17:28.778005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.777969    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"
Oct 30 21:17:28.868589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.868536    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:17:28.896554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.886335    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92d5d744-5c548bc679-rtwfr"]
Oct 30 21:17:28.910551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:28.909799    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:17:29.010555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.010456    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\" (UniqueName: \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\") pod \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") "
Oct 30 21:17:29.017557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.011021    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\") pod \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\" (UID: \"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9\") "
Oct 30 21:17:29.017557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.011410    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume" pod "0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" (UID: "0ed16483-21f2-478c-8e92-f7c2ff2fa7c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:29.017557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.011807    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:29.024223 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.024163    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q" pod "0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" (UID: "0ed16483-21f2-478c-8e92-f7c2ff2fa7c9"). InnerVolumeSpecName "kube-api-access-hw67q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:29.114373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.114286    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw67q\" (UniqueName: \"kubernetes.io/projected/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9-kube-api-access-hw67q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:29.794552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.793905    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"
Oct 30 21:17:29.794552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.794248    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db" event={"ID":"0ed16483-21f2-478c-8e92-f7c2ff2fa7c9","Type":"ContainerDied","Data":"ff14399b9f59772d6cb30fefd642f1c528df4953fb76d12cadb0b0bf5108b340"}
Oct 30 21:17:29.794552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.794317    7037 scope.go:122] "RemoveContainer" containerID="bcd6ba6c05b7f0a3de84cbeaf7772237959ca8d5675d347f290e9e0a1b572cf8"
Oct 30 21:17:29.861026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.860980    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:17:29.877797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:29.877761    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b62455c-7b9674c54c-mf2db"]
Oct 30 21:17:30.031563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:30.031518    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"]
Oct 30 21:17:30.040064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:30.040013    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e70d4f-4sd68"]
Oct 30 21:17:30.714547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:30.712521    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" path="/var/lib/kubelet/pods/0ed16483-21f2-478c-8e92-f7c2ff2fa7c9/volumes"
Oct 30 21:17:30.714547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:30.713343    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1" path="/var/lib/kubelet/pods/33fd9a45-5f13-43ec-a6e3-0e705b9dcfa1/volumes"
Oct 30 21:17:30.714547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:30.713993    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e37cbe-e571-4237-b688-5a3e6086aaed" path="/var/lib/kubelet/pods/36e37cbe-e571-4237-b688-5a3e6086aaed/volumes"
Oct 30 21:17:31.044822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.044770    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"]
Oct 30 21:17:31.057208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.057169    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca3a2156-mxzxx"]
Oct 30 21:17:31.278630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.278598    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:17:31.390255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390219    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390270    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390343    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390387    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390532    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390608    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390687    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390702    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:31.390921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390807    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\") pod \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\" (UID: \"c08d9d3a-72b9-4467-88a0-80cfd04496c9\") "
Oct 30 21:17:31.390980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.390927    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:31.391122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391086    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:31.391215 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391198    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:31.391386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391304    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:31.391740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391719    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.391816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391744    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.391816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391758    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.391816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391770    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.391816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.391783    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.393041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.393010    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:31.393462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.393438    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:31.393651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.393621    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "kube-api-access-g95cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:31.394083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.394057    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo" pod "c08d9d3a-72b9-4467-88a0-80cfd04496c9" (UID: "c08d9d3a-72b9-4467-88a0-80cfd04496c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:17:31.493659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.493583    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.493659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.493617    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95cd\" (UniqueName: \"kubernetes.io/projected/c08d9d3a-72b9-4467-88a0-80cfd04496c9-kube-api-access-g95cd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.493659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.493633    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.493659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.493647    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d9d3a-72b9-4467-88a0-80cfd04496c9-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:17:31.841945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.841889    7037 generic.go:356] "Generic (PLEG): container finished" podID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerID="1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd" exitCode=0
Oct 30 21:17:31.842338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.842303    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" event={"ID":"c08d9d3a-72b9-4467-88a0-80cfd04496c9","Type":"ContainerDied","Data":"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"}
Oct 30 21:17:31.842573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.842547    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f" event={"ID":"c08d9d3a-72b9-4467-88a0-80cfd04496c9","Type":"ContainerDied","Data":"0b7a979b49a3e32c28518cd35a5be21d58e1c790e2e26a909a443d89bd4b6885"}
Oct 30 21:17:31.842755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.842730    7037 scope.go:122] "RemoveContainer" containerID="1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"
Oct 30 21:17:31.843098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.843081    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"
Oct 30 21:17:31.877251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.877086    7037 scope.go:122] "RemoveContainer" containerID="1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"
Oct 30 21:17:31.877648 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:17:31.877590    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd\": container with ID starting with 1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd not found: ID does not exist" containerID="1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"
Oct 30 21:17:31.877766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.877663    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd"} err="failed to get container status \"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd\": rpc error: code = NotFound desc = could not find container \"1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd\": container with ID starting with 1d86eaecf13830dd970033fad67fd809838041ce6c5cf4a18407f468c021ddcd not found: ID does not exist"
Oct 30 21:17:31.916666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.916634    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:17:31.922134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:31.922099    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-dd1e164d-gw-openshift-default-86c9fdcd9-fmz9f"]
Oct 30 21:17:32.710406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.710365    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f9e857-5b49-4a3f-9f54-58f86c34214f" path="/var/lib/kubelet/pods/47f9e857-5b49-4a3f-9f54-58f86c34214f/volumes"
Oct 30 21:17:32.711023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.711004    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" path="/var/lib/kubelet/pods/c08d9d3a-72b9-4467-88a0-80cfd04496c9/volumes"
Oct 30 21:17:32.735468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.735429    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:17:32.736576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736548    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerName="istio-proxy"
Oct 30 21:17:32.736576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736574    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerName="istio-proxy"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736591    7037 cpu_manager.go:414] "containerMap: removing container" podUID="aad46b49-95ef-4dc4-860d-8af9ef564059" containerName="payload-processing"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736599    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad46b49-95ef-4dc4-860d-8af9ef564059" containerName="payload-processing"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736613    7037 cpu_manager.go:414] "containerMap: removing container" podUID="36e37cbe-e571-4237-b688-5a3e6086aaed" containerName="payload-pre-processing"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736620    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e37cbe-e571-4237-b688-5a3e6086aaed" containerName="payload-pre-processing"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736657    7037 cpu_manager.go:414] "containerMap: removing container" podUID="331f13a5-3319-44f1-8d43-298ff083e4f7" containerName="revoke-keys"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736664    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331f13a5-3319-44f1-8d43-298ff083e4f7" containerName="revoke-keys"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736686    7037 cpu_manager.go:414] "containerMap: removing container" podUID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" containerName="maas-api"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736693    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" containerName="maas-api"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736727    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" containerName="payload-pre-processing"
Oct 30 21:17:32.736770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736734    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" containerName="payload-pre-processing"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736908    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08d9d3a-72b9-4467-88a0-80cfd04496c9" containerName="istio-proxy"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736922    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad46b49-95ef-4dc4-860d-8af9ef564059" containerName="payload-processing"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736931    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e37cbe-e571-4237-b688-5a3e6086aaed" containerName="payload-pre-processing"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736944    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="331f13a5-3319-44f1-8d43-298ff083e4f7" containerName="revoke-keys"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736958    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="30368a71-4282-4cbd-a4a1-1a3b812ab39f" containerName="maas-api"
Oct 30 21:17:32.737267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.736970    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed16483-21f2-478c-8e92-f7c2ff2fa7c9" containerName="payload-pre-processing"
Oct 30 21:17:32.741558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.741477    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.759062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.759037    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:17:32.763684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.763661    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-399049f3-dockercfg-wlgdl\""
Oct 30 21:17:32.763809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.763749    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-399049f3\""
Oct 30 21:17:32.824240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.824201    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.824411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.824326    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qjd\" (UniqueName: \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.944718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.927231    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.944718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.927343    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qjd\" (UniqueName: \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.944718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.928710    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:32.983617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:32.976920    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qjd\" (UniqueName: \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\") pod \"payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:33.074595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.074548    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"]
Oct 30 21:17:33.078308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.078279    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:33.088375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.088338    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c5e1af-rljms"]
Oct 30 21:17:33.490069 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:17:33.490031    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774e6dc0_36f4_45e2_bd92_bdf5c0857134.slice/crio-d038818ffa210d00ed5286af2a05899554ac716b6dd85cb08a54533270fbf33f WatchSource:0}: Error finding container d038818ffa210d00ed5286af2a05899554ac716b6dd85cb08a54533270fbf33f: Status 404 returned error can't find the container with id d038818ffa210d00ed5286af2a05899554ac716b6dd85cb08a54533270fbf33f
Oct 30 21:17:33.492258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.492235    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:17:33.674956 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:17:33.674915    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:17:33.866416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.866378    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" event={"ID":"774e6dc0-36f4-45e2-bd92-bdf5c0857134","Type":"ContainerStarted","Data":"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"}
Oct 30 21:17:33.866876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.866436    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" event={"ID":"774e6dc0-36f4-45e2-bd92-bdf5c0857134","Type":"ContainerStarted","Data":"d038818ffa210d00ed5286af2a05899554ac716b6dd85cb08a54533270fbf33f"}
Oct 30 21:17:33.866876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.866460    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:33.910343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:33.910293    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" podStartSLOduration=1.91027729 podStartE2EDuration="1.91027729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:33.907016771 +0000 UTC m=+997.318556209" watchObservedRunningTime="2026-10-30 21:17:33.91027729 +0000 UTC m=+997.321816726"
Oct 30 21:17:34.710280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:34.710243    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4729b2-7727-4c12-8607-f07ae756a7b5" path="/var/lib/kubelet/pods/0e4729b2-7727-4c12-8607-f07ae756a7b5/volumes"
Oct 30 21:17:36.077128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.077045    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"]
Oct 30 21:17:36.084307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.084276    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e2cb04-2mk68"]
Oct 30 21:17:36.442087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.442014    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:17:36.446736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.446711    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.464183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.464153    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c906b806\""
Oct 30 21:17:36.467846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.467820    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:17:36.494356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.494319    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqsv\" (UniqueName: \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.494551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.494448    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.522778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.522743    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:17:36.526894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.526872    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.547093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.547055    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:17:36.613511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.601762    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqsv\" (UniqueName: \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.613511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.601867    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.613511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.602139    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.613511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.602230    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltm5\" (UniqueName: \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.625608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.623688    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c906b806-dockercfg-mvklx\""
Oct 30 21:17:36.652514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.652040    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.692563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.681658    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqsv\" (UniqueName: \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\") pod \"payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") " pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.728632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.728594    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltm5\" (UniqueName: \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.729248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.729222    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.732510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.730138    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.749553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.748566    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0520c35c-a699-4e8f-9fe0-b1816c41100b" path="/var/lib/kubelet/pods/0520c35c-a699-4e8f-9fe0-b1816c41100b/volumes"
Oct 30 21:17:36.750238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.750216    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltm5\" (UniqueName: \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\") pod \"payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:36.761254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.761177    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:36.844022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:36.843978    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:37.101746 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:17:37.101711    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f70912_23f1_408f_9030_13ffcd2cccab.slice/crio-d220609cd431ee8d9ab94ac35d8fea6989ba352e2607aec8aabf502615d892a6 WatchSource:0}: Error finding container d220609cd431ee8d9ab94ac35d8fea6989ba352e2607aec8aabf502615d892a6: Status 404 returned error can't find the container with id d220609cd431ee8d9ab94ac35d8fea6989ba352e2607aec8aabf502615d892a6
Oct 30 21:17:37.104179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:37.104154    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:17:37.178280 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:17:37.178209    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a3baa0_b984_4f5d_87cc_67b3484ae3d1.slice/crio-88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768 WatchSource:0}: Error finding container 88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768: Status 404 returned error can't find the container with id 88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768
Oct 30 21:17:37.180502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:37.180457    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:17:37.984513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:37.975808    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" event={"ID":"92a3baa0-b984-4f5d-87cc-67b3484ae3d1","Type":"ContainerStarted","Data":"dd427115b23a20919297e956798f5281bb38870b98f11022b03b14b4138a3f5c"}
Oct 30 21:17:37.984513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:37.975914    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" event={"ID":"92a3baa0-b984-4f5d-87cc-67b3484ae3d1","Type":"ContainerStarted","Data":"88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768"}
Oct 30 21:17:38.004537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.004109    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:38.055663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.041091    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" podStartSLOduration=2.041069134 podStartE2EDuration="2.041069134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:38.023957741 +0000 UTC m=+1001.435497180" watchObservedRunningTime="2026-10-30 21:17:38.041069134 +0000 UTC m=+1001.452608572"
Oct 30 21:17:38.055663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.046674    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" event={"ID":"e2f70912-23f1-408f-9030-13ffcd2cccab","Type":"ContainerStarted","Data":"4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7"}
Oct 30 21:17:38.055663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.055335    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:38.055663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.055382    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" event={"ID":"e2f70912-23f1-408f-9030-13ffcd2cccab","Type":"ContainerStarted","Data":"d220609cd431ee8d9ab94ac35d8fea6989ba352e2607aec8aabf502615d892a6"}
Oct 30 21:17:38.091078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:38.090899    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" podStartSLOduration=2.090883154 podStartE2EDuration="2.090883154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:38.089983311 +0000 UTC m=+1001.501522749" watchObservedRunningTime="2026-10-30 21:17:38.090883154 +0000 UTC m=+1001.502422590"
Oct 30 21:17:42.976515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:42.973169    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:17:43.208516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.198216    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:43.247577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.246613    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:43.294642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.294593    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:43.397720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.397629    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:17:43.491212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.491170    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:43.523280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:43.523178    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:17:44.038838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:44.038799    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"]
Oct 30 21:17:44.048299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:44.048262    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06686839-fwdrn"]
Oct 30 21:17:44.714012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:44.713975    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba1d493-76a0-4bd6-89f4-058f021c506e" path="/var/lib/kubelet/pods/5ba1d493-76a0-4bd6-89f4-058f021c506e/volumes"
Oct 30 21:17:44.886727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:44.886695    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:17:46.032715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:46.032664    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"]
Oct 30 21:17:46.045028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:46.044993    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebcd5343-2g5h6"]
Oct 30 21:17:46.709668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:46.709634    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6357c3-063e-43b7-b923-88876d359f2a" path="/var/lib/kubelet/pods/bd6357c3-063e-43b7-b923-88876d359f2a/volumes"
Oct 30 21:17:49.060292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:49.060265    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:17:50.074084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:50.074043    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:17:57.037038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:57.037006    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"]
Oct 30 21:17:57.041679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:57.041651    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5894c8-twx76"]
Oct 30 21:17:58.709868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:17:58.709826    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab1522b-86d3-420a-a9ea-4ac3213e507a" path="/var/lib/kubelet/pods/4ab1522b-86d3-420a-a9ea-4ac3213e507a/volumes"
Oct 30 21:18:03.674535 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:18:03.674461    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:18:04.029479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:04.029393    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"]
Oct 30 21:18:04.041799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:04.041764    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83d73c08-8xrvw"]
Oct 30 21:18:04.712266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:04.712221    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da452ee6-fd5c-4938-a3b0-78924406360e" path="/var/lib/kubelet/pods/da452ee6-fd5c-4938-a3b0-78924406360e/volumes"
Oct 30 21:18:12.361888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.361851    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"]
Oct 30 21:18:12.369447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.369424    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.390991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.390959    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:12.391134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.391076    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:18:12.394925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.394894    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"]
Oct 30 21:18:12.440370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.440336    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.440565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.440511    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.440628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.440615    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8h76\" (UniqueName: \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.541936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.541897    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.542131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.542012    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.542131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.542109    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8h76\" (UniqueName: \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.542719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.542695    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.544336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.544313    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.561997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.561972    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8h76\" (UniqueName: \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\") pod \"maas-api-revoke-keys-e2e-ait-399049f3-r4skd\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:12.710330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:12.710247    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:13.020861 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:18:13.020830    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c86ddc1_36c7_4850_9682_5d3e6a6c79ba.slice/crio-bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4 WatchSource:0}: Error finding container bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4: Status 404 returned error can't find the container with id bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4
Oct 30 21:18:13.025343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:13.025319    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"]
Oct 30 21:18:13.543841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:13.543804    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd" event={"ID":"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba","Type":"ContainerStarted","Data":"deae3dcabb1fe4c23f9593f2a1c8db9f14703bee87b4ef7e0fe92a2853077301"}
Oct 30 21:18:13.544244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:13.543872    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd" event={"ID":"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba","Type":"ContainerStarted","Data":"bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4"}
Oct 30 21:18:13.586880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:13.586778    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd" podStartSLOduration=1.586764333 podStartE2EDuration="1.586764333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:13.582332051 +0000 UTC m=+1036.993871502" watchObservedRunningTime="2026-10-30 21:18:13.586764333 +0000 UTC m=+1036.998303769"
Oct 30 21:18:14.554322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:14.554290    7037 generic.go:356] "Generic (PLEG): container finished" podID="8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" containerID="deae3dcabb1fe4c23f9593f2a1c8db9f14703bee87b4ef7e0fe92a2853077301" exitCode=0
Oct 30 21:18:14.554322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:14.554327    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd" event={"ID":"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba","Type":"ContainerDied","Data":"deae3dcabb1fe4c23f9593f2a1c8db9f14703bee87b4ef7e0fe92a2853077301"}
Oct 30 21:18:15.742867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.742840    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:15.898587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.898550    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\") pod \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") "
Oct 30 21:18:15.898790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.898624    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\") pod \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") "
Oct 30 21:18:15.898790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.898732    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\" (UniqueName: \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\") pod \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\" (UID: \"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba\") "
Oct 30 21:18:15.900651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.900618    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca" pod "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" (UID: "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:15.902777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.902745    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76" pod "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" (UID: "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba"). InnerVolumeSpecName "kube-api-access-k8h76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:15.903593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:15.903549    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca" pod "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" (UID: "8c86ddc1-36c7-4850-9682-5d3e6a6c79ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:16.000828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.000797    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:16.000828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.000824    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:16.000828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.000835    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8h76\" (UniqueName: \"kubernetes.io/projected/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba-kube-api-access-k8h76\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:16.573097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.573069    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"
Oct 30 21:18:16.573296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.573099    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd" event={"ID":"8c86ddc1-36c7-4850-9682-5d3e6a6c79ba","Type":"ContainerDied","Data":"bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4"}
Oct 30 21:18:16.573296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:16.573146    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff9667370693cfb182c83096a21c5a42a3b83debf302e378e981392b3705aa4"
Oct 30 21:18:21.008516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.007685    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.106514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.106233    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:18:21.191632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.191585    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.263705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.263602    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.297643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.297591    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.396641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.396590    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.483764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.483720    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:21.551559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:21.550452    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 21:18:22.843976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:22.843936    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:18:24.092512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:24.083173    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"]
Oct 30 21:18:24.106546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:24.103398    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03b7a9c-fnmx6"]
Oct 30 21:18:24.710199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:24.710154    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f7bffa-842c-400a-a270-d384b12cf1d1" path="/var/lib/kubelet/pods/62f7bffa-842c-400a-a270-d384b12cf1d1/volumes"
Oct 30 21:18:25.467938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:25.467904    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:18:25.499586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:25.499552    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:18:26.446827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:26.446777    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"]
Oct 30 21:18:26.459005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:26.458977    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399049f3-r4skd"]
Oct 30 21:18:26.711749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:26.711671    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" path="/var/lib/kubelet/pods/8c86ddc1-36c7-4850-9682-5d3e6a6c79ba/volumes"
Oct 30 21:18:27.849751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:27.849576    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" podUID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" containerName="payload-pre-processing" containerID="cri-o://f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3" gracePeriod=25
Oct 30 21:18:28.346620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.346591    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:18:28.429112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.429076    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\" (UniqueName: \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\") pod \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") "
Oct 30 21:18:28.429626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.429593    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\") pod \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\" (UID: \"774e6dc0-36f4-45e2-bd92-bdf5c0857134\") "
Oct 30 21:18:28.429987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.429963    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume" pod "774e6dc0-36f4-45e2-bd92-bdf5c0857134" (UID: "774e6dc0-36f4-45e2-bd92-bdf5c0857134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:28.430408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.430381    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774e6dc0-36f4-45e2-bd92-bdf5c0857134-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:28.431390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.431362    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd" pod "774e6dc0-36f4-45e2-bd92-bdf5c0857134" (UID: "774e6dc0-36f4-45e2-bd92-bdf5c0857134"). InnerVolumeSpecName "kube-api-access-l5qjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:28.531658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.531577    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qjd\" (UniqueName: \"kubernetes.io/projected/774e6dc0-36f4-45e2-bd92-bdf5c0857134-kube-api-access-l5qjd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:28.861214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.861180    7037 generic.go:356] "Generic (PLEG): container finished" podID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" containerID="f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3" exitCode=0
Oct 30 21:18:28.861698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.861248    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"
Oct 30 21:18:28.861698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.861264    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" event={"ID":"774e6dc0-36f4-45e2-bd92-bdf5c0857134","Type":"ContainerDied","Data":"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"}
Oct 30 21:18:28.861698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.861327    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl" event={"ID":"774e6dc0-36f4-45e2-bd92-bdf5c0857134","Type":"ContainerDied","Data":"d038818ffa210d00ed5286af2a05899554ac716b6dd85cb08a54533270fbf33f"}
Oct 30 21:18:28.861698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.861354    7037 scope.go:122] "RemoveContainer" containerID="f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"
Oct 30 21:18:28.884483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.884462    7037 scope.go:122] "RemoveContainer" containerID="f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"
Oct 30 21:18:28.884828 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:18:28.884804    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3\": container with ID starting with f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3 not found: ID does not exist" containerID="f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"
Oct 30 21:18:28.884883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.884840    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3"} err="failed to get container status \"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3\": rpc error: code = NotFound desc = could not find container \"f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3\": container with ID starting with f920c00bf2f6fd6cb34d7025ee7ea54cc30c1df8884cba729cdb8937cac846b3 not found: ID does not exist"
Oct 30 21:18:28.917767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.917736    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:18:28.931573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:28.931538    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-399049f3-77f856b49f-f24sl"]
Oct 30 21:18:30.468253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.468213    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" podUID="e2f70912-23f1-408f-9030-13ffcd2cccab" containerName="payload-processing" containerID="cri-o://4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7" gracePeriod=25
Oct 30 21:18:30.504895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.504838    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" podUID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" containerName="payload-pre-processing" containerID="cri-o://dd427115b23a20919297e956798f5281bb38870b98f11022b03b14b4138a3f5c" gracePeriod=25
Oct 30 21:18:30.593064 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:18:30.592898    7037 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f70912_23f1_408f_9030_13ffcd2cccab.slice/crio-4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f70912_23f1_408f_9030_13ffcd2cccab.slice/crio-conmon-4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:18:30.749556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.724788    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" path="/var/lib/kubelet/pods/774e6dc0-36f4-45e2-bd92-bdf5c0857134/volumes"
Oct 30 21:18:30.903510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.903449    7037 generic.go:356] "Generic (PLEG): container finished" podID="e2f70912-23f1-408f-9030-13ffcd2cccab" containerID="4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7" exitCode=0
Oct 30 21:18:30.903962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.903910    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" event={"ID":"e2f70912-23f1-408f-9030-13ffcd2cccab","Type":"ContainerDied","Data":"4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7"}
Oct 30 21:18:30.906431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.906406    7037 generic.go:356] "Generic (PLEG): container finished" podID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" containerID="dd427115b23a20919297e956798f5281bb38870b98f11022b03b14b4138a3f5c" exitCode=0
Oct 30 21:18:30.906698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.906676    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" event={"ID":"92a3baa0-b984-4f5d-87cc-67b3484ae3d1","Type":"ContainerDied","Data":"dd427115b23a20919297e956798f5281bb38870b98f11022b03b14b4138a3f5c"}
Oct 30 21:18:30.906871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.906842    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72" event={"ID":"92a3baa0-b984-4f5d-87cc-67b3484ae3d1","Type":"ContainerDied","Data":"88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768"}
Oct 30 21:18:30.907015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.906998    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88a4e43ca4206203cbc813a3464a9c795e5f3206ec288d089c92bbb97844e768"
Oct 30 21:18:30.931727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.931694    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:18:30.964115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.964085    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:18:30.994692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.994655    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\") pod \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") "
Oct 30 21:18:30.994911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.994895    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\" (UniqueName: \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\") pod \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\" (UID: \"92a3baa0-b984-4f5d-87cc-67b3484ae3d1\") "
Oct 30 21:18:30.995711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.995047    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume" pod "92a3baa0-b984-4f5d-87cc-67b3484ae3d1" (UID: "92a3baa0-b984-4f5d-87cc-67b3484ae3d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:30.995711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.995434    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:30.997213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:30.997182    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5" pod "92a3baa0-b984-4f5d-87cc-67b3484ae3d1" (UID: "92a3baa0-b984-4f5d-87cc-67b3484ae3d1"). InnerVolumeSpecName "kube-api-access-tltm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:31.096457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.096423    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\" (UniqueName: \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\") pod \"e2f70912-23f1-408f-9030-13ffcd2cccab\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") "
Oct 30 21:18:31.096754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.096732    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\") pod \"e2f70912-23f1-408f-9030-13ffcd2cccab\" (UID: \"e2f70912-23f1-408f-9030-13ffcd2cccab\") "
Oct 30 21:18:31.097196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.097164    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume" pod "e2f70912-23f1-408f-9030-13ffcd2cccab" (UID: "e2f70912-23f1-408f-9030-13ffcd2cccab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:31.097439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.097418    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f70912-23f1-408f-9030-13ffcd2cccab-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:31.097439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.097440    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tltm5\" (UniqueName: \"kubernetes.io/projected/92a3baa0-b984-4f5d-87cc-67b3484ae3d1-kube-api-access-tltm5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:31.098768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.098735    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv" pod "e2f70912-23f1-408f-9030-13ffcd2cccab" (UID: "e2f70912-23f1-408f-9030-13ffcd2cccab"). InnerVolumeSpecName "kube-api-access-znqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:31.199420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.199383    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znqsv\" (UniqueName: \"kubernetes.io/projected/e2f70912-23f1-408f-9030-13ffcd2cccab-kube-api-access-znqsv\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:31.916380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.916348    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"
Oct 30 21:18:31.916380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.916374    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq" event={"ID":"e2f70912-23f1-408f-9030-13ffcd2cccab","Type":"ContainerDied","Data":"d220609cd431ee8d9ab94ac35d8fea6989ba352e2607aec8aabf502615d892a6"}
Oct 30 21:18:31.916932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.916443    7037 scope.go:122] "RemoveContainer" containerID="4d4f5db7863c401fefe2001247bcf0f83773e7345d9cab15c967cb8053b665a7"
Oct 30 21:18:31.916932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.916554    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"
Oct 30 21:18:31.995658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:31.995616    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:18:32.006803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:32.006766    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c906b806-9477cb87f-bgv72"]
Oct 30 21:18:32.033152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:32.033115    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:18:32.037943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:32.037912    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c906b806-6f7d69cd6d-kmczq"]
Oct 30 21:18:32.709256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:32.709227    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" path="/var/lib/kubelet/pods/92a3baa0-b984-4f5d-87cc-67b3484ae3d1/volumes"
Oct 30 21:18:32.709699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:32.709684    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f70912-23f1-408f-9030-13ffcd2cccab" path="/var/lib/kubelet/pods/e2f70912-23f1-408f-9030-13ffcd2cccab/volumes"
Oct 30 21:18:33.683821 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:18:33.683772    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:18:42.079741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.079707    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"]
Oct 30 21:18:42.080446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080429    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" containerName="revoke-keys"
Oct 30 21:18:42.080446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080445    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" containerName="revoke-keys"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080471    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e2f70912-23f1-408f-9030-13ffcd2cccab" containerName="payload-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080477    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f70912-23f1-408f-9030-13ffcd2cccab" containerName="payload-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080504    7037 cpu_manager.go:414] "containerMap: removing container" podUID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" containerName="payload-pre-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080510    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" containerName="payload-pre-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080546    7037 cpu_manager.go:414] "containerMap: removing container" podUID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" containerName="payload-pre-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080551    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" containerName="payload-pre-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080654    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="774e6dc0-36f4-45e2-bd92-bdf5c0857134" containerName="payload-pre-processing"
Oct 30 21:18:42.080662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080667    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c86ddc1-36c7-4850-9682-5d3e6a6c79ba" containerName="revoke-keys"
Oct 30 21:18:42.081022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080676    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f70912-23f1-408f-9030-13ffcd2cccab" containerName="payload-processing"
Oct 30 21:18:42.081022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.080685    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3baa0-b984-4f5d-87cc-67b3484ae3d1" containerName="payload-pre-processing"
Oct 30 21:18:42.084306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.084287    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.104297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.104267    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"]
Oct 30 21:18:42.110306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.110283    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:18:42.110306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.110288    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:42.261142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.261108    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.261321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.261224    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.261443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.261430    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qjh\" (UniqueName: \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.363056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.362962    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.363056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.363043    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.363298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.363211    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qjh\" (UniqueName: \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.363807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.363783    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.365425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.365400    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.383208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.383182    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qjh\" (UniqueName: \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\") pod \"maas-api-revoke-sub-08d021d0-5mxxq\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.398059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.398036    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:42.696767 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:18:42.696739    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef8662b_d830_4e0f_af16_472c3439944e.slice/crio-d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68 WatchSource:0}: Error finding container d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68: Status 404 returned error can't find the container with id d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68
Oct 30 21:18:42.698965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:42.698941    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"]
Oct 30 21:18:43.028298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:43.028266    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq" event={"ID":"3ef8662b-d830-4e0f-af16-472c3439944e","Type":"ContainerStarted","Data":"d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68"}
Oct 30 21:18:44.039308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:44.039204    7037 generic.go:356] "Generic (PLEG): container finished" podID="3ef8662b-d830-4e0f-af16-472c3439944e" containerID="3e35aeb1ad2c2596f65b07f1f2babcc5fa248a26c17fbf427cbe4b70bf6a2b1d" exitCode=0
Oct 30 21:18:44.039779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:44.039311    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq" event={"ID":"3ef8662b-d830-4e0f-af16-472c3439944e","Type":"ContainerDied","Data":"3e35aeb1ad2c2596f65b07f1f2babcc5fa248a26c17fbf427cbe4b70bf6a2b1d"}
Oct 30 21:18:45.206792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.206767    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:45.228507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.228462    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\" (UniqueName: \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\") pod \"3ef8662b-d830-4e0f-af16-472c3439944e\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") "
Oct 30 21:18:45.228756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.228733    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\") pod \"3ef8662b-d830-4e0f-af16-472c3439944e\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") "
Oct 30 21:18:45.228861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.228846    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\") pod \"3ef8662b-d830-4e0f-af16-472c3439944e\" (UID: \"3ef8662b-d830-4e0f-af16-472c3439944e\") "
Oct 30 21:18:45.229199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.229175    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca" pod "3ef8662b-d830-4e0f-af16-472c3439944e" (UID: "3ef8662b-d830-4e0f-af16-472c3439944e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:45.229457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.229440    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:45.230572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.230540    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh" pod "3ef8662b-d830-4e0f-af16-472c3439944e" (UID: "3ef8662b-d830-4e0f-af16-472c3439944e"). InnerVolumeSpecName "kube-api-access-h2qjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:45.230677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.230580    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca" pod "3ef8662b-d830-4e0f-af16-472c3439944e" (UID: "3ef8662b-d830-4e0f-af16-472c3439944e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:45.330953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.330923    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qjh\" (UniqueName: \"kubernetes.io/projected/3ef8662b-d830-4e0f-af16-472c3439944e-kube-api-access-h2qjh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:45.330953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:45.330951    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef8662b-d830-4e0f-af16-472c3439944e-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:18:46.059418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:46.059387    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"
Oct 30 21:18:46.059611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:46.059419    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq" event={"ID":"3ef8662b-d830-4e0f-af16-472c3439944e","Type":"ContainerDied","Data":"d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68"}
Oct 30 21:18:46.059611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:46.059473    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41dd11452dbdfadd8f06a6367fb63dfdb32ea0c8efd679b50b6c839c6419d68"
Oct 30 21:18:56.976180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:56.976144    7037 scope.go:122] "RemoveContainer" containerID="af473f46255baeaa950f6149bc61a291662d51ca9939635fe2f857531b2c1e38"
Oct 30 21:18:56.993795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:56.993766    7037 scope.go:122] "RemoveContainer" containerID="722a7747e4cee7c447f8eec793d9c326f6ae1a661365f7815079814fe0dc6eb2"
Oct 30 21:18:57.004768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.004740    7037 scope.go:122] "RemoveContainer" containerID="6cac683d804c04cf683dc6c6ff1e4d918e9e9a3e2bf23b08bb9162ed80a8c15c"
Oct 30 21:18:57.016698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.016673    7037 scope.go:122] "RemoveContainer" containerID="d830563fe20411be26f6eaaba07dc7b0fd03b3b50fa8927e70f3350884b3dc4b"
Oct 30 21:18:57.026693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.026673    7037 scope.go:122] "RemoveContainer" containerID="d7dc177b188cd8014f7ef1fbc703270b32003dc598d3a7aa5b8400642c36fcfb"
Oct 30 21:18:57.043932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.043897    7037 scope.go:122] "RemoveContainer" containerID="e15191ac87aae42d0b230b69991508c0d9db117df92f66071f2a7c5e8c9c3fe6"
Oct 30 21:18:57.063913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.063891    7037 scope.go:122] "RemoveContainer" containerID="fddb844a9d7899a2b901ce580a91e9a9b455b38d25c5860d006d73ce719311f4"
Oct 30 21:18:57.074450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.074429    7037 scope.go:122] "RemoveContainer" containerID="bad95b5a7c16d166ccf7cbfc55f2c8d7bcb1b621ee150be7e601d5e04ae721d3"
Oct 30 21:18:57.084480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.084459    7037 scope.go:122] "RemoveContainer" containerID="da7df1dd708aec01c2b5057cfbaefc16d1437378af19584dd53d298af87deb87"
Oct 30 21:18:57.094221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.094197    7037 scope.go:122] "RemoveContainer" containerID="1b08cd21ecf3ba7de796f4afa82f0af1d17c0dd63bafa20ae55be39265c8d401"
Oct 30 21:18:57.104249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.104225    7037 scope.go:122] "RemoveContainer" containerID="5a6801f42f6be6edae22bd8d7f8e35ec268b04879a0496d3c9016ffb6d3f77ab"
Oct 30 21:18:57.113715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.113693    7037 scope.go:122] "RemoveContainer" containerID="5a81fcb702de281f6176c4d8fc671de86f7b0fe966810be0e2f942e30d96654d"
Oct 30 21:18:57.122962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:18:57.122932    7037 scope.go:122] "RemoveContainer" containerID="f707f654805d4bdb12be25086051b47c66018126c15866c30f8cd9914bdf9b4b"
Oct 30 21:19:03.674241 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:19:03.674203    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:19:11.449101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:11.448854    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:11.507514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:11.502111    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:11.593823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:11.591031    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:11.656527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:11.633726    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.010669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.001828    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.179580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.179547    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"]
Oct 30 21:19:12.240535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.239584    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.256633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.255624    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d088bc-2gl9r"]
Oct 30 21:19:12.256633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.255719    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.316547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.316108    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.317125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.316816    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.319035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.317315    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:19:12.474978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.472588    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.474978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.473192    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:19:12.595002 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.593043    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.595002 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.593673    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:19:12.641532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.637667    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.641532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.638274    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:19:12.641532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.641008    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:12.734657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.734469    7037 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 30 21:19:12.760157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.760122    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018a1388-b8f3-4ad0-945d-4ec326d4b56c" path="/var/lib/kubelet/pods/018a1388-b8f3-4ad0-945d-4ec326d4b56c/volumes"
Oct 30 21:19:12.761045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.761019    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:19:12.763789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.763761    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:19:12.816009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:12.813274    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:19:14.071237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.071202    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"]
Oct 30 21:19:14.071993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.071977    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3ef8662b-d830-4e0f-af16-472c3439944e" containerName="revoke-keys"
Oct 30 21:19:14.072060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.071995    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef8662b-d830-4e0f-af16-472c3439944e" containerName="revoke-keys"
Oct 30 21:19:14.072152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.072142    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef8662b-d830-4e0f-af16-472c3439944e" containerName="revoke-keys"
Oct 30 21:19:14.077793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.077768    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.088021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.087994    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"]
Oct 30 21:19:14.107211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.107185    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:14.107354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.107216    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:19:14.192873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.192842    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.193048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.192893    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lpn\" (UniqueName: \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.193140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.193124    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.294761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.294731    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.294956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.294937    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.295014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.294976    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lpn\" (UniqueName: \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.295717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.295686    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.297470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.297446    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.314838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.314809    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lpn\" (UniqueName: \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\") pod \"maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.398219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.398130    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:14.702760 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:19:14.702726    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35451ee6_d13e_4fcf_811b_e5b3398f66b0.slice/crio-7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc WatchSource:0}: Error finding container 7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc: Status 404 returned error can't find the container with id 7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc
Oct 30 21:19:14.711895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.711868    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"]
Oct 30 21:19:14.769619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:14.769577    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx" event={"ID":"35451ee6-d13e-4fcf-811b-e5b3398f66b0","Type":"ContainerStarted","Data":"7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc"}
Oct 30 21:19:15.780642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:15.780609    7037 generic.go:356] "Generic (PLEG): container finished" podID="35451ee6-d13e-4fcf-811b-e5b3398f66b0" containerID="110e68f3c25097cd4b03a58843d1a0aff63fa5ac7a828fab6ef3e096cb175b60" exitCode=0
Oct 30 21:19:15.781039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:15.780689    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx" event={"ID":"35451ee6-d13e-4fcf-811b-e5b3398f66b0","Type":"ContainerDied","Data":"110e68f3c25097cd4b03a58843d1a0aff63fa5ac7a828fab6ef3e096cb175b60"}
Oct 30 21:19:16.965654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:16.965627    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:17.045437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.045345    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\") pod \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") "
Oct 30 21:19:17.045437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.045387    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\") pod \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") "
Oct 30 21:19:17.045437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.045414    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\" (UniqueName: \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\") pod \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\" (UID: \"35451ee6-d13e-4fcf-811b-e5b3398f66b0\") "
Oct 30 21:19:17.045870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.045842    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca" pod "35451ee6-d13e-4fcf-811b-e5b3398f66b0" (UID: "35451ee6-d13e-4fcf-811b-e5b3398f66b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:17.046118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.046095    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:19:17.048235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.048203    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca" pod "35451ee6-d13e-4fcf-811b-e5b3398f66b0" (UID: "35451ee6-d13e-4fcf-811b-e5b3398f66b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:17.048349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.048317    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn" pod "35451ee6-d13e-4fcf-811b-e5b3398f66b0" (UID: "35451ee6-d13e-4fcf-811b-e5b3398f66b0"). InnerVolumeSpecName "kube-api-access-j2lpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:17.147701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.147669    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35451ee6-d13e-4fcf-811b-e5b3398f66b0-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:19:17.147701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.147698    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2lpn\" (UniqueName: \"kubernetes.io/projected/35451ee6-d13e-4fcf-811b-e5b3398f66b0-kube-api-access-j2lpn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:19:17.798410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.798381    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"
Oct 30 21:19:17.798648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.798418    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx" event={"ID":"35451ee6-d13e-4fcf-811b-e5b3398f66b0","Type":"ContainerDied","Data":"7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc"}
Oct 30 21:19:17.798648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:17.798466    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7325edbaf2c0ddf600f8154c6b94668d458e2f0b10cb3d80ac508dab9400e1cc"
Oct 30 21:19:33.172059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.172023    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:19:33.172805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.172785    7037 cpu_manager.go:414] "containerMap: removing container" podUID="35451ee6-d13e-4fcf-811b-e5b3398f66b0" containerName="revoke-keys"
Oct 30 21:19:33.172863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.172809    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35451ee6-d13e-4fcf-811b-e5b3398f66b0" containerName="revoke-keys"
Oct 30 21:19:33.173013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.173003    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="35451ee6-d13e-4fcf-811b-e5b3398f66b0" containerName="revoke-keys"
Oct 30 21:19:33.177105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.177084    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.204545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.200954    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c46397-serving-cert\""
Oct 30 21:19:33.212564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.212010    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:19:33.269605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.269558    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.269817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.269716    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.270076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.270036    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6vn\" (UniqueName: \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.270330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.270303    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.371800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.371767    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6vn\" (UniqueName: \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.371985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.371893    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.371985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.371914    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.371985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.371956    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.372718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.372688    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.374637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.374610    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.374757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.374640    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.394875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.394849    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6vn\" (UniqueName: \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\") pod \"maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.525347 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.524190    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:33.716942 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:19:33.716872    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:19:33.882531 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:19:33.882459    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e3b69_f5d2_4bc9_95b8_ebf11602e7c0.slice/crio-e11285174fa7856d703ce3b569d0550b307a1507f8dda33a09100654cd189bbb WatchSource:0}: Error finding container e11285174fa7856d703ce3b569d0550b307a1507f8dda33a09100654cd189bbb: Status 404 returned error can't find the container with id e11285174fa7856d703ce3b569d0550b307a1507f8dda33a09100654cd189bbb
Oct 30 21:19:33.884211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.884185    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:19:33.950664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:33.950626    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" event={"ID":"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0","Type":"ContainerStarted","Data":"e11285174fa7856d703ce3b569d0550b307a1507f8dda33a09100654cd189bbb"}
Oct 30 21:19:34.960139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:34.960102    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" event={"ID":"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0","Type":"ContainerStarted","Data":"fe33c1a803b3df522a9f0bea26fc6c3276370b14112ea682373d366db9c9e87a"}
Oct 30 21:19:34.960608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:34.960215    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:35.009012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:35.008968    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" podStartSLOduration=1.699734021 podStartE2EDuration="2.00895546s" totalImagesPullingTime="309.221439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:35.003733267 +0000 UTC m=+1118.415272703" watchObservedRunningTime="2026-10-30 21:19:35.00895546 +0000 UTC m=+1118.420494926"
Oct 30 21:19:35.030009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:35.029977    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"]
Oct 30 21:19:35.034231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:35.034209    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbd05764-qlb2k"]
Oct 30 21:19:36.709946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:36.709906    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7909e148-60bf-4f69-9dc9-857230751c39" path="/var/lib/kubelet/pods/7909e148-60bf-4f69-9dc9-857230751c39/volumes"
Oct 30 21:19:38.650034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:38.649986    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"]
Oct 30 21:19:38.669851 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:38.669812    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1f954ca9-2kjvx"]
Oct 30 21:19:38.728201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:38.727979    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35451ee6-d13e-4fcf-811b-e5b3398f66b0" path="/var/lib/kubelet/pods/35451ee6-d13e-4fcf-811b-e5b3398f66b0/volumes"
Oct 30 21:19:40.984330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:40.984297    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:19:47.030929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:47.030885    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"]
Oct 30 21:19:47.055944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:47.055900    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae50bfe-pc6np"]
Oct 30 21:19:48.712371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:48.712331    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551bc85c-a40c-4a30-9227-c28f0bde1819" path="/var/lib/kubelet/pods/551bc85c-a40c-4a30-9227-c28f0bde1819/volumes"
Oct 30 21:19:57.234312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:57.234279    7037 scope.go:122] "RemoveContainer" containerID="f99cc3af037247eb6de8e13c73c2b615991398d11ac553dd6eae54eba158c28b"
Oct 30 21:19:57.246579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:57.246554    7037 scope.go:122] "RemoveContainer" containerID="9777c4d57d6cab4ba6b4af0d45e372671f5198982157d8485756147daaaf0c5d"
Oct 30 21:19:59.032730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:59.032698    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"]
Oct 30 21:19:59.044585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:19:59.044557    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602075ad-rlsnz"]
Oct 30 21:20:00.710711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:00.710677    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65e59b5-1012-49ac-a0f8-5ec66c09c9ac" path="/var/lib/kubelet/pods/f65e59b5-1012-49ac-a0f8-5ec66c09c9ac/volumes"
Oct 30 21:20:03.681987 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:20:03.681953    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:20:06.029591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:06.029561    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"]
Oct 30 21:20:06.041357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:06.041328    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28bd106-rg6xz"]
Oct 30 21:20:06.708815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:06.708781    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c87b2a6-6e19-41eb-b1eb-95d28be054cd" path="/var/lib/kubelet/pods/0c87b2a6-6e19-41eb-b1eb-95d28be054cd/volumes"
Oct 30 21:20:12.037452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:12.037412    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"]
Oct 30 21:20:12.049427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:12.049388    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5a6726-tb2s6"]
Oct 30 21:20:12.708730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:12.708699    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1166de44-05d3-4773-b5ed-7e19eedaee71" path="/var/lib/kubelet/pods/1166de44-05d3-4773-b5ed-7e19eedaee71/volumes"
Oct 30 21:20:14.592396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.592362    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:20:14.597242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.597210    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.630193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.630165    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ad73a4\""
Oct 30 21:20:14.630193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.630171    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ad73a4-dockercfg-546z4\""
Oct 30 21:20:14.665087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.665060    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:20:14.835802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.751136    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.842680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.842586    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff26d\" (UniqueName: \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.911465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.911429    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:20:14.924730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.924696    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:20:14.937634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.929715    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:14.950885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.950856    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.951076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.951063    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff26d\" (UniqueName: \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.952718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.952688    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:14.974334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:14.974300    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff26d\" (UniqueName: \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\") pod \"payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:15.053132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.053089    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdf42\" (UniqueName: \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.053320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.053188    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.155533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.155425    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdf42\" (UniqueName: \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.155724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.155543    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.156527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.156481    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.176455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.176428    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdf42\" (UniqueName: \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\") pod \"payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.212083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.212053    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:15.262950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.262916    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:15.515858 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:20:15.515827    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea76e62d_ce9f_4f42_8a7f_c7e8ddd1314f.slice/crio-5f58702d67e75a4be9020ec73409e25d5149f5cf0ca5829d282f302bfdb501e2 WatchSource:0}: Error finding container 5f58702d67e75a4be9020ec73409e25d5149f5cf0ca5829d282f302bfdb501e2: Status 404 returned error can't find the container with id 5f58702d67e75a4be9020ec73409e25d5149f5cf0ca5829d282f302bfdb501e2
Oct 30 21:20:15.517166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.517141    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:20:15.592767 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:20:15.592739    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27835da_2169_4319_aad7_bf998a74c2e0.slice/crio-cbfc574a9ca094ca4bac9db52f6b88aa79de386f7792cae2623222a8c9a2adc8 WatchSource:0}: Error finding container cbfc574a9ca094ca4bac9db52f6b88aa79de386f7792cae2623222a8c9a2adc8: Status 404 returned error can't find the container with id cbfc574a9ca094ca4bac9db52f6b88aa79de386f7792cae2623222a8c9a2adc8
Oct 30 21:20:15.595737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.595710    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:20:15.804464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.804378    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:20:15.809090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.809066    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.824403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.824366    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1da2183-serving-cert\""
Oct 30 21:20:15.831899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.831872    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:20:15.871384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.871338    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.871637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.871511    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdfh\" (UniqueName: \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.871637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.871581    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.871948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.871921    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.973600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.973562    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.973600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.973679    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.974214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.973813    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdfh\" (UniqueName: \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.974214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.973867    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.974405 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:20:15.974249    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-serving-cert: secret "maas-api-e2e-mt-c1da2183-serving-cert" not found
Oct 30 21:20:15.974405 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:20:15.974316    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls podName:a1807b61-e196-46de-b287-0fc7ce0f42bb nodeName:}" failed. No retries permitted until 2026-10-30 21:20:16.474296684 +0000 UTC m=+1159.885836114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls") pod "maas-api-e2e-mt-c1da2183-66768659c7-whcq7" (UID: "a1807b61-e196-46de-b287-0fc7ce0f42bb") : secret "maas-api-e2e-mt-c1da2183-serving-cert" not found
Oct 30 21:20:15.974405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.974341    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.977120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.977094    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:15.995834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:15.995808    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdfh\" (UniqueName: \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:16.354499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.354448    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" event={"ID":"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f","Type":"ContainerStarted","Data":"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"}
Oct 30 21:20:16.354709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.354553    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" event={"ID":"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f","Type":"ContainerStarted","Data":"5f58702d67e75a4be9020ec73409e25d5149f5cf0ca5829d282f302bfdb501e2"}
Oct 30 21:20:16.354709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.354586    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:16.356226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.356195    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" event={"ID":"f27835da-2169-4319-aad7-bf998a74c2e0","Type":"ContainerStarted","Data":"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"}
Oct 30 21:20:16.356370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.356243    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" event={"ID":"f27835da-2169-4319-aad7-bf998a74c2e0","Type":"ContainerStarted","Data":"cbfc574a9ca094ca4bac9db52f6b88aa79de386f7792cae2623222a8c9a2adc8"}
Oct 30 21:20:16.356370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.356264    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:16.398844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.398799    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" podStartSLOduration=2.398784982 podStartE2EDuration="2.398784982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.395484775 +0000 UTC m=+1159.807024211" watchObservedRunningTime="2026-10-30 21:20:16.398784982 +0000 UTC m=+1159.810324419"
Oct 30 21:20:16.437538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.437497    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" podStartSLOduration=2.437471918 podStartE2EDuration="2.437471918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.434886798 +0000 UTC m=+1159.846426234" watchObservedRunningTime="2026-10-30 21:20:16.437471918 +0000 UTC m=+1159.849011353"
Oct 30 21:20:16.481053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.481021    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:16.483482 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.483459    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-whcq7\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:16.731112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:16.731029    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:17.034601 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:20:17.034573    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1807b61_e196_46de_b287_0fc7ce0f42bb.slice/crio-7633c2068febfad53a5ef0631f4ab1b3b745c54b6ec298c6e81faabdb5d5e1fc WatchSource:0}: Error finding container 7633c2068febfad53a5ef0631f4ab1b3b745c54b6ec298c6e81faabdb5d5e1fc: Status 404 returned error can't find the container with id 7633c2068febfad53a5ef0631f4ab1b3b745c54b6ec298c6e81faabdb5d5e1fc
Oct 30 21:20:17.037552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:17.037533    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:20:17.368436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:17.368400    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" event={"ID":"a1807b61-e196-46de-b287-0fc7ce0f42bb","Type":"ContainerStarted","Data":"7633c2068febfad53a5ef0631f4ab1b3b745c54b6ec298c6e81faabdb5d5e1fc"}
Oct 30 21:20:18.027852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.027821    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"]
Oct 30 21:20:18.042195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.042163    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64816519-2bg66"]
Oct 30 21:20:18.378102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.378066    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" event={"ID":"a1807b61-e196-46de-b287-0fc7ce0f42bb","Type":"ContainerStarted","Data":"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"}
Oct 30 21:20:18.378327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.378143    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:18.419997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.419935    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" podStartSLOduration=3.056400945 podStartE2EDuration="3.41991985s" totalImagesPullingTime="363.518905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:18.416312486 +0000 UTC m=+1161.827851921" watchObservedRunningTime="2026-10-30 21:20:18.41991985 +0000 UTC m=+1161.831459286"
Oct 30 21:20:18.710945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:18.710863    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8df5461-967b-430a-8ba4-f91c33b97f00" path="/var/lib/kubelet/pods/d8df5461-967b-430a-8ba4-f91c33b97f00/volumes"
Oct 30 21:20:24.391757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:24.391723    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:20:25.031754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:25.031718    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"]
Oct 30 21:20:25.043844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:25.043815    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a059beda-fbx8b"]
Oct 30 21:20:26.711141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:26.711105    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228a0698-53eb-42da-9a1e-513b6f9673c9" path="/var/lib/kubelet/pods/228a0698-53eb-42da-9a1e-513b6f9673c9/volumes"
Oct 30 21:20:27.027781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:27.027707    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"]
Oct 30 21:20:27.039505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:27.039463    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87b0cf2-cdcfh"]
Oct 30 21:20:27.371798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:27.371772    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:20:27.371981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:27.371833    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:20:28.711084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:28.711054    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2aa6ec-0950-4882-a906-f4c7afd67c94" path="/var/lib/kubelet/pods/ae2aa6ec-0950-4882-a906-f4c7afd67c94/volumes"
Oct 30 21:20:32.033133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:32.033097    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"]
Oct 30 21:20:32.045360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:32.045319    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7facfa59-9nbdg"]
Oct 30 21:20:32.714119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:32.714085    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927d44d3-7995-439a-99ab-601e46672b93" path="/var/lib/kubelet/pods/927d44d3-7995-439a-99ab-601e46672b93/volumes"
Oct 30 21:20:33.673911 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:20:33.673867    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:20:44.031458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:44.031361    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"]
Oct 30 21:20:44.042434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:44.042403    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108f89c6-pf92p"]
Oct 30 21:20:44.709038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:44.709006    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2bbf29-b9e4-4894-a351-b210f0b6b18e" path="/var/lib/kubelet/pods/1e2bbf29-b9e4-4894-a351-b210f0b6b18e/volumes"
Oct 30 21:20:47.029706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:47.029668    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"]
Oct 30 21:20:47.047389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:47.047344    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b069a2-kl4k7"]
Oct 30 21:20:48.721926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:48.721887    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f3637-5828-4d8f-845f-ae9a0f2170c8" path="/var/lib/kubelet/pods/216f3637-5828-4d8f-845f-ae9a0f2170c8/volumes"
Oct 30 21:20:53.033452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:53.033416    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"]
Oct 30 21:20:53.043636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:53.043599    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a2332f-f467s"]
Oct 30 21:20:54.139414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:54.139380    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"]
Oct 30 21:20:54.143159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:54.143132    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc0c627-6ptzx"]
Oct 30 21:20:54.721929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:54.721891    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080a587b-5ab2-4824-b347-289433a7fc29" path="/var/lib/kubelet/pods/080a587b-5ab2-4824-b347-289433a7fc29/volumes"
Oct 30 21:20:54.722700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:54.722675    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e023f8b7-94dd-4f07-9baa-ab5c62ad33dd" path="/var/lib/kubelet/pods/e023f8b7-94dd-4f07-9baa-ab5c62ad33dd/volumes"
Oct 30 21:20:57.037995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.037956    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"]
Oct 30 21:20:57.050746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.050713    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70afd3c7-p9zqx"]
Oct 30 21:20:57.283873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.283844    7037 scope.go:122] "RemoveContainer" containerID="88bce672de8f766ab2868ec7e36d4643da24b9732bd42d0f3b7b26b2a4618632"
Oct 30 21:20:57.297770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.297744    7037 scope.go:122] "RemoveContainer" containerID="741a90b9a0b3776b33be85cebc4377254fc2668e6a538aab0575595d92e0e75c"
Oct 30 21:20:57.320443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.320418    7037 scope.go:122] "RemoveContainer" containerID="55a56b09f5bf41e1ba62200dd8a6646328231e0fc5f354f257cf037492232a80"
Oct 30 21:20:57.338828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:57.338703    7037 scope.go:122] "RemoveContainer" containerID="9b943fd2dc0ce3dddd961377e565e4c7b4466632bfa1b43019964f080f271f88"
Oct 30 21:20:58.723322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:20:58.723286    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c9960a-5b82-44e1-aa8d-186f284d66f7" path="/var/lib/kubelet/pods/65c9960a-5b82-44e1-aa8d-186f284d66f7/volumes"
Oct 30 21:21:03.673024 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:21:03.672985    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:21:04.974902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:04.974856    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:21:04.982367 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:04.982347    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.019537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.014042    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-70d3a9\""
Oct 30 21:21:05.026609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.026576    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:21:05.031561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.031535    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:21:05.037965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.037939    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:21:05.038097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.038052    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.053442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.053420    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-70d3a9-dockercfg-czp7d\""
Oct 30 21:21:05.121924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.121894    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.122108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.122087    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwnv\" (UniqueName: \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.122173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.122122    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86bf4\" (UniqueName: \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.122364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.122346    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.225566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.225422    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.225787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.225662    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.225869 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.225836    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwnv\" (UniqueName: \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.225937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.225869    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bf4\" (UniqueName: \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.228518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.227167    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.228518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.227908    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.291628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.290089    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bf4\" (UniqueName: \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\") pod \"payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.291628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.290513    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwnv\" (UniqueName: \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\") pod \"payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.321312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.321276    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.355455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.355416    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.689145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.689108    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:21:05.695583 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:05.694835    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696591a5_faa5_4462_9994_292c599c6030.slice/crio-e760329335b735261b4288dfc96ff8e3022f9e919d5e1e998b95aef2a15b06e2 WatchSource:0}: Error finding container e760329335b735261b4288dfc96ff8e3022f9e919d5e1e998b95aef2a15b06e2: Status 404 returned error can't find the container with id e760329335b735261b4288dfc96ff8e3022f9e919d5e1e998b95aef2a15b06e2
Oct 30 21:21:05.709777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.709750    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:21:05.714659 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:05.714633    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e24018a_efff_4258_aebd_b3722015e5d3.slice/crio-44adeb047d450d47fc82d41d5181c031c213cdaa13f9bda15c1adfcf1a889b33 WatchSource:0}: Error finding container 44adeb047d450d47fc82d41d5181c031c213cdaa13f9bda15c1adfcf1a889b33: Status 404 returned error can't find the container with id 44adeb047d450d47fc82d41d5181c031c213cdaa13f9bda15c1adfcf1a889b33
Oct 30 21:21:05.881190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.881155    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" event={"ID":"9e24018a-efff-4258-aebd-b3722015e5d3","Type":"ContainerStarted","Data":"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"}
Oct 30 21:21:05.881356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.881226    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:05.881356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.881252    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" event={"ID":"9e24018a-efff-4258-aebd-b3722015e5d3","Type":"ContainerStarted","Data":"44adeb047d450d47fc82d41d5181c031c213cdaa13f9bda15c1adfcf1a889b33"}
Oct 30 21:21:05.883513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.883468    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" event={"ID":"696591a5-faa5-4462-9994-292c599c6030","Type":"ContainerStarted","Data":"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"}
Oct 30 21:21:05.883654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.883535    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" event={"ID":"696591a5-faa5-4462-9994-292c599c6030","Type":"ContainerStarted","Data":"e760329335b735261b4288dfc96ff8e3022f9e919d5e1e998b95aef2a15b06e2"}
Oct 30 21:21:05.883654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.883570    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:05.928083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.928031    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" podStartSLOduration=1.928017036 podStartE2EDuration="1.928017036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:05.923793591 +0000 UTC m=+1209.335333030" watchObservedRunningTime="2026-10-30 21:21:05.928017036 +0000 UTC m=+1209.339556511"
Oct 30 21:21:05.966174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:05.966068    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" podStartSLOduration=1.9660545109999998 podStartE2EDuration="1.966054511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:05.963441085 +0000 UTC m=+1209.374980521" watchObservedRunningTime="2026-10-30 21:21:05.966054511 +0000 UTC m=+1209.377594019"
Oct 30 21:21:13.034543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:13.034506    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"]
Oct 30 21:21:13.043987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:13.043960    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793863bc-pw5hc"]
Oct 30 21:21:14.709241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:14.709197    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e546b6-0de7-43b2-8a74-f98d50884b6f" path="/var/lib/kubelet/pods/b3e546b6-0de7-43b2-8a74-f98d50884b6f/volumes"
Oct 30 21:21:16.177443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.177405    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:21:16.182268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.182246    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.200567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.199147    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-59fb17\""
Oct 30 21:21:16.225471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.219804    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:21:16.246140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.246103    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:21:16.252186 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.252157    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.268203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.268168    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.268410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.268256    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvtl\" (UniqueName: \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.268567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.268475    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.268567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.268547    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz5s\" (UniqueName: \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.273993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.273968    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-59fb17-dockercfg-98f9d\""
Oct 30 21:21:16.290813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.290705    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:21:16.370270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.370233    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.370434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.370292    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz5s\" (UniqueName: \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.370434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.370414    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.370530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.370464    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvtl\" (UniqueName: \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.371051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.371025    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.371190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.371157    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.389866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.389836    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvtl\" (UniqueName: \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\") pod \"payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") " pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.389999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.389955    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz5s\" (UniqueName: \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\") pod \"payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.503636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.503542    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:16.596746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.596704    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:16.905513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.899594    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:21:16.905809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.905748    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:21:16.915994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:16.915966    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:21:17.009337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.009290    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:21:17.146316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.146275    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" event={"ID":"5fdbb835-aabf-4b41-a4ea-297b60a0f30d","Type":"ContainerStarted","Data":"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"}
Oct 30 21:21:17.146541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.146342    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" event={"ID":"5fdbb835-aabf-4b41-a4ea-297b60a0f30d","Type":"ContainerStarted","Data":"b54a8cec02d5e2fdc7056381f519d44af706c94a8a35dddf364382b01d4ff6b2"}
Oct 30 21:21:17.146541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.146399    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:17.148325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.148296    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" event={"ID":"fa44567f-a2ad-4f81-aeb1-1c3669bedb69","Type":"ContainerStarted","Data":"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"}
Oct 30 21:21:17.148455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.148343    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" event={"ID":"fa44567f-a2ad-4f81-aeb1-1c3669bedb69","Type":"ContainerStarted","Data":"3e743fb9621a04ef4704eb94f55144a7e4eee3b9167ecca009f0ef2415e2fb7d"}
Oct 30 21:21:17.148455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.148425    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:17.190273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.190213    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" podStartSLOduration=1.190195123 podStartE2EDuration="1.190195123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:17.186331061 +0000 UTC m=+1220.597870499" watchObservedRunningTime="2026-10-30 21:21:17.190195123 +0000 UTC m=+1220.601734562"
Oct 30 21:21:17.230633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:17.230571    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" podStartSLOduration=1.230553432 podStartE2EDuration="1.230553432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:17.226632314 +0000 UTC m=+1220.638171752" watchObservedRunningTime="2026-10-30 21:21:17.230553432 +0000 UTC m=+1220.642092870"
Oct 30 21:21:18.143971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.143901    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:21:18.149294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.149271    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.166507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.166464    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:21:18.173507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.173471    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0e7510-dockercfg-pfsn2\""
Oct 30 21:21:18.173663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.173471    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0e7510\""
Oct 30 21:21:18.367956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.367895    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.371828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.368700    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztbr\" (UniqueName: \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.375517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.375481    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:21:18.392179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.392139    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.400991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.400867    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:21:18.478819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.478776    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.479059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.478937    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.479207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.479138    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64np\" (UniqueName: \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.479256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.479201    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztbr\" (UniqueName: \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.480971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.480948    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.499462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.499437    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztbr\" (UniqueName: \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\") pod \"payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:18.580871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.580839    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64np\" (UniqueName: \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.581158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.581136    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.581951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.581925    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.601144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.601113    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64np\" (UniqueName: \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\") pod \"payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") " pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.727343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.727263    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:18.768724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:18.768695    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:19.030486 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.030401    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"]
Oct 30 21:21:19.034427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.034396    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303f3994-lb7qx"]
Oct 30 21:21:19.038805 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:19.038771    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b297685_d673_4246_82c2_61a0c073d078.slice/crio-ee6ac210f273b10150340a8206dd2b1a63e7b127e1a941fe3737c9639ec5abcd WatchSource:0}: Error finding container ee6ac210f273b10150340a8206dd2b1a63e7b127e1a941fe3737c9639ec5abcd: Status 404 returned error can't find the container with id ee6ac210f273b10150340a8206dd2b1a63e7b127e1a941fe3737c9639ec5abcd
Oct 30 21:21:19.042621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.042600    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:21:19.095621 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:19.095593    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bd36e3_c428_4568_aa2c_4711a3990f85.slice/crio-9e7906ab452cc1d6c2a55d39644fe902d1034a131374c0575eafd4665843c887 WatchSource:0}: Error finding container 9e7906ab452cc1d6c2a55d39644fe902d1034a131374c0575eafd4665843c887: Status 404 returned error can't find the container with id 9e7906ab452cc1d6c2a55d39644fe902d1034a131374c0575eafd4665843c887
Oct 30 21:21:19.099242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.099221    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:21:19.185160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.185119    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" event={"ID":"7b297685-d673-4246-82c2-61a0c073d078","Type":"ContainerStarted","Data":"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"}
Oct 30 21:21:19.185317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.185222    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:19.185317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.185249    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" event={"ID":"7b297685-d673-4246-82c2-61a0c073d078","Type":"ContainerStarted","Data":"ee6ac210f273b10150340a8206dd2b1a63e7b127e1a941fe3737c9639ec5abcd"}
Oct 30 21:21:19.187011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.186976    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" event={"ID":"48bd36e3-c428-4568-aa2c-4711a3990f85","Type":"ContainerStarted","Data":"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"}
Oct 30 21:21:19.187135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.187030    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" event={"ID":"48bd36e3-c428-4568-aa2c-4711a3990f85","Type":"ContainerStarted","Data":"9e7906ab452cc1d6c2a55d39644fe902d1034a131374c0575eafd4665843c887"}
Oct 30 21:21:19.187135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.187079    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:19.227425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.227354    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" podStartSLOduration=1.227342408 podStartE2EDuration="1.227342408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:19.223556565 +0000 UTC m=+1222.635096002" watchObservedRunningTime="2026-10-30 21:21:19.227342408 +0000 UTC m=+1222.638881861"
Oct 30 21:21:19.265057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:19.265008    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" podStartSLOduration=1.2649949249999999 podStartE2EDuration="1.264994925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:19.263923497 +0000 UTC m=+1222.675462933" watchObservedRunningTime="2026-10-30 21:21:19.264994925 +0000 UTC m=+1222.676534361"
Oct 30 21:21:20.709847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:20.709816    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a318d2-0089-43bd-ade7-7d98e1fc68db" path="/var/lib/kubelet/pods/46a318d2-0089-43bd-ade7-7d98e1fc68db/volumes"
Oct 30 21:21:21.289716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.289682    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:21:21.294956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.294934    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.314887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.314849    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:21:21.322473 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.322448    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a4782a\""
Oct 30 21:21:21.324168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.324147    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a4782a-dockercfg-t4clj\""
Oct 30 21:21:21.433226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.433174    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5vp\" (UniqueName: \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.433388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.433268    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.534755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.534717    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5vp\" (UniqueName: \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.535200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.534820    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.537868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.536111    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.554876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.554794    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5vp\" (UniqueName: \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\") pod \"payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.611435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.611400    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:21.939090 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:21.939059    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cab665c_5805_43da_a21b_095dbad3bdf7.slice/crio-1a62226fbb8ca2dee450eb61cf209f0b093ccc70a64f87e20f06b9842faaf30f WatchSource:0}: Error finding container 1a62226fbb8ca2dee450eb61cf209f0b093ccc70a64f87e20f06b9842faaf30f: Status 404 returned error can't find the container with id 1a62226fbb8ca2dee450eb61cf209f0b093ccc70a64f87e20f06b9842faaf30f
Oct 30 21:21:21.940427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:21.940403    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:21:22.215597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:22.215503    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" event={"ID":"0cab665c-5805-43da-a21b-095dbad3bdf7","Type":"ContainerStarted","Data":"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"}
Oct 30 21:21:22.215597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:22.215574    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:22.215597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:22.215593    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" event={"ID":"0cab665c-5805-43da-a21b-095dbad3bdf7","Type":"ContainerStarted","Data":"1a62226fbb8ca2dee450eb61cf209f0b093ccc70a64f87e20f06b9842faaf30f"}
Oct 30 21:21:22.258430 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:22.258378    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" podStartSLOduration=1.25836439 podStartE2EDuration="1.25836439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:22.254615149 +0000 UTC m=+1225.666154599" watchObservedRunningTime="2026-10-30 21:21:22.25836439 +0000 UTC m=+1225.669903827"
Oct 30 21:21:27.699347 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.699313    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:21:27.704269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.704242    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.725721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.725674    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-70d3a9-serving-cert\""
Oct 30 21:21:27.735250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.735219    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:21:27.760654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.760615    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.760820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.760690    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.760885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.760838    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.761199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.761174    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsfl\" (UniqueName: \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.863307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.863268    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsfl\" (UniqueName: \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.863705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.863629    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.864820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.863710    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.864820 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:21:27.863871    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-serving-cert: secret "maas-api-e2e-shared-b-w3-70d3a9-serving-cert" not found
Oct 30 21:21:27.864820 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:21:27.863943    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls podName:ab275431-00d5-4a8f-8de4-f780fde9d792 nodeName:}" failed. No retries permitted until 2026-10-30 21:21:28.36392148 +0000 UTC m=+1231.775460909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls") pod "maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" (UID: "ab275431-00d5-4a8f-8de4-f780fde9d792") : secret "maas-api-e2e-shared-b-w3-70d3a9-serving-cert" not found
Oct 30 21:21:27.864820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.863967    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.868076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.864985    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.868250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.868224    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:27.885242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:27.885171    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsfl\" (UniqueName: \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:28.161047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.161018    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:21:28.161339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.161317    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:21:28.373196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.373157    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:28.376267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.376230    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:28.625116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.625084    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:28.980164 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.980132    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:21:28.981819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:28.981796    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:21:29.052537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:29.052456    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"]
Oct 30 21:21:29.055418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:29.055389    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31068716-c7ncp"]
Oct 30 21:21:29.308750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:29.308547    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" event={"ID":"ab275431-00d5-4a8f-8de4-f780fde9d792","Type":"ContainerStarted","Data":"d40935ee4cd302175680669bf6d2c3ce1a760e80d178af1c98ef87a1267d2b72"}
Oct 30 21:21:30.199238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.199205    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:21:30.199670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.199652    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:21:30.321099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.321059    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" event={"ID":"ab275431-00d5-4a8f-8de4-f780fde9d792","Type":"ContainerStarted","Data":"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"}
Oct 30 21:21:30.321482 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.321448    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:30.366753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.366701    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" podStartSLOduration=3.004240661 podStartE2EDuration="3.366686701s" totalImagesPullingTime="362.44604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:30.362680026 +0000 UTC m=+1233.774219463" watchObservedRunningTime="2026-10-30 21:21:30.366686701 +0000 UTC m=+1233.778226137"
Oct 30 21:21:30.710276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:30.710245    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c105dd7-4a76-4488-bc9f-5c0fddc65791" path="/var/lib/kubelet/pods/9c105dd7-4a76-4488-bc9f-5c0fddc65791/volumes"
Oct 30 21:21:33.228358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:33.228331    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:21:33.674950 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:21:33.674907    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:21:36.334611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:36.334581    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:21:37.287548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.287517    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:21:37.293904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.293876    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.311691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.311662    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:21:37.318613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.318585    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-59fb17-serving-cert\""
Oct 30 21:21:37.324520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.324463    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7hf\" (UniqueName: \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.324691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.324609    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.324757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.324724    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.324757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.324747    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.425937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.425907    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7hf\" (UniqueName: \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.426360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.426009    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.426360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.426105    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.426360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.426123    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.426749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.426716    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.428532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.428479    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.428624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.428557    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.445536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.445477    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7hf\" (UniqueName: \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\") pod \"maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.612363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.612330    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:37.931059 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:21:37.931028    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f860b90_ffa6_431c_af8f_b3e1cb6c7599.slice/crio-126ef6f54aed0daeeef3348e969115af4cecbcbc2d26b910490cd5eccabc3314 WatchSource:0}: Error finding container 126ef6f54aed0daeeef3348e969115af4cecbcbc2d26b910490cd5eccabc3314: Status 404 returned error can't find the container with id 126ef6f54aed0daeeef3348e969115af4cecbcbc2d26b910490cd5eccabc3314
Oct 30 21:21:37.932711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:37.932683    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:21:38.390441 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:38.390397    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" event={"ID":"3f860b90-ffa6-431c-af8f-b3e1cb6c7599","Type":"ContainerStarted","Data":"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"}
Oct 30 21:21:38.390601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:38.390464    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" event={"ID":"3f860b90-ffa6-431c-af8f-b3e1cb6c7599","Type":"ContainerStarted","Data":"126ef6f54aed0daeeef3348e969115af4cecbcbc2d26b910490cd5eccabc3314"}
Oct 30 21:21:38.390601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:38.390542    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:38.434387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:38.434319    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" podStartSLOduration=1.079163167 podStartE2EDuration="1.434300523s" totalImagesPullingTime="355.137356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:38.433543422 +0000 UTC m=+1241.845082860" watchObservedRunningTime="2026-10-30 21:21:38.434300523 +0000 UTC m=+1241.845839962"
Oct 30 21:21:44.404753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:44.404724    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:21:57.408521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.408466    7037 scope.go:122] "RemoveContainer" containerID="5e50aef54b12cafdfde0097007675e23619a255720bfaefbb2ab7cbdd4b4f463"
Oct 30 21:21:57.422674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.422645    7037 scope.go:122] "RemoveContainer" containerID="13f0e9ae2c5ce869f29a7607c70ca3d827b12353a8432ff0d1190de13336a98d"
Oct 30 21:21:57.433379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.433353    7037 scope.go:122] "RemoveContainer" containerID="507f1458599240f2e38ddd72bb5c1135e3b7b5b793e33b786303b33eb8988435"
Oct 30 21:21:57.443465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.443445    7037 scope.go:122] "RemoveContainer" containerID="2a99344824c20a29d9460df03b73078f417edea9b018780bf2a80b9cec7f0334"
Oct 30 21:21:57.453706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.453684    7037 scope.go:122] "RemoveContainer" containerID="265f3fdb14de058a329bfb3767394d46063f7db6f6ff24f56af17edb12b20178"
Oct 30 21:21:57.463597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.463573    7037 scope.go:122] "RemoveContainer" containerID="bf878934b9251bccc75099da97daae3cc22f62bfeaee07c8f05f98729fdd1ef8"
Oct 30 21:21:57.473391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.473372    7037 scope.go:122] "RemoveContainer" containerID="5cc2b25420f27a5dddfbbcfad7ce3ff664a5f651c40e70902b65e14a8cb0f46b"
Oct 30 21:21:57.483304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.483283    7037 scope.go:122] "RemoveContainer" containerID="f611868bbe59533f4f1d288a732cb2fdaf1f07256005dd1df17e9843a1cf5dd6"
Oct 30 21:21:57.492826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.492808    7037 scope.go:122] "RemoveContainer" containerID="af260e57a5101e6b86d9b5a69d281e54363f7d488776d8e4f665aa3b52534c43"
Oct 30 21:21:57.502326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.502304    7037 scope.go:122] "RemoveContainer" containerID="f807e6134d463fadd3ab871b277098130e3bbc5c715d5ad64e480cc86d64990c"
Oct 30 21:21:57.511899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:21:57.511880    7037 scope.go:122] "RemoveContainer" containerID="5596bd61c22402f12b6fd006aab3dfef6e6e4ee318e88ac296b401cbfdacf76b"
Oct 30 21:22:03.708869 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:03.708825    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:22:05.036790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:05.036755    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"]
Oct 30 21:22:05.048990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:05.048960    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91666752-99cvk"]
Oct 30 21:22:06.709073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:06.708990    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4" path="/var/lib/kubelet/pods/700f0f3e-c359-4c1f-ad0f-8b3cca0be1e4/volumes"
Oct 30 21:22:17.087710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.087644    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:22:17.088214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.087926    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy" containerID="cri-o://640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a" gracePeriod=30
Oct 30 21:22:17.098163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.095259    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:22:17.098163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.095572    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy" containerID="cri-o://0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e" gracePeriod=30
Oct 30 21:22:17.105142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.104729    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:22:17.105142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.104959    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy" containerID="cri-o://50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8" gracePeriod=30
Oct 30 21:22:17.112091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.112046    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:22:17.112328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.112295    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy" containerID="cri-o://20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0" gracePeriod=30
Oct 30 21:22:17.125019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.124805    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:22:17.125264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.125237    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" containerID="cri-o://e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5" gracePeriod=30
Oct 30 21:22:17.134673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.134618    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:22:17.134934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:17.134888    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy" containerID="cri-o://be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21" gracePeriod=30
Oct 30 21:22:18.474790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.474754    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:22:18.480943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.480919    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.496092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.496062    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4266b7-dockercfg-bk5m5\""
Oct 30 21:22:18.498953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.498927    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:22:18.505444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.505383    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4266b7\""
Oct 30 21:22:18.525300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.525266    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:22:18.530793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.530758    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.553036 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.553003    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:22:18.649630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.649576    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.650031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.649960    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.650267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.650248    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbps\" (UniqueName: \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.650530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.650477    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzv9\" (UniqueName: \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.753146    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.753213    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.753295    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbps\" (UniqueName: \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.753389    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzv9\" (UniqueName: \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.754382    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.758557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.755668    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.785786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.785115    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbps\" (UniqueName: \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\") pod \"payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") " pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.788111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.788080    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzv9\" (UniqueName: \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\") pod \"payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:18.828565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.810262    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:18.857564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:18.854736    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:19.268936 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:19.268858    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c9d4b6_c5fd_4bfd_931e_596a5f37dc5c.slice/crio-a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d WatchSource:0}: Error finding container a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d: Status 404 returned error can't find the container with id a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d
Oct 30 21:22:19.271312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.271284    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:22:19.299210 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:19.299183    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0283276a_024e_4243_8b0b_fd1381fb1da2.slice/crio-62db3b5e52f902ac2b9fa266d33a8e032f2e3037b1afcba8af75d153ff0cd811 WatchSource:0}: Error finding container 62db3b5e52f902ac2b9fa266d33a8e032f2e3037b1afcba8af75d153ff0cd811: Status 404 returned error can't find the container with id 62db3b5e52f902ac2b9fa266d33a8e032f2e3037b1afcba8af75d153ff0cd811
Oct 30 21:22:19.301365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.301343    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:22:19.923726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.923691    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" event={"ID":"0283276a-024e-4243-8b0b-fd1381fb1da2","Type":"ContainerStarted","Data":"fa43d17599ccbc5b202b3344a92bd0f6b1d87fd690b4176c2dccba1f4a703229"}
Oct 30 21:22:19.924190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.923745    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" event={"ID":"0283276a-024e-4243-8b0b-fd1381fb1da2","Type":"ContainerStarted","Data":"62db3b5e52f902ac2b9fa266d33a8e032f2e3037b1afcba8af75d153ff0cd811"}
Oct 30 21:22:19.924617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.924581    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:19.926788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.926759    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" event={"ID":"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c","Type":"ContainerStarted","Data":"3cdd2f842a7c0df9bf2e752b0f55081299f4b83f820d04a28bb813d80a4ab3f2"}
Oct 30 21:22:19.926953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.926810    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:19.926953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.926827    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" event={"ID":"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c","Type":"ContainerStarted","Data":"a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d"}
Oct 30 21:22:19.970472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:19.970418    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" podStartSLOduration=1.9704036710000001 podStartE2EDuration="1.970403671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:19.967773725 +0000 UTC m=+1283.379313168" watchObservedRunningTime="2026-10-30 21:22:19.970403671 +0000 UTC m=+1283.381943107"
Oct 30 21:22:20.015800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.015750    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" podStartSLOduration=2.015734744 podStartE2EDuration="2.015734744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:20.010248657 +0000 UTC m=+1283.421788085" watchObservedRunningTime="2026-10-30 21:22:20.015734744 +0000 UTC m=+1283.427274200"
Oct 30 21:22:20.094665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.094633    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:22:20.099499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.099470    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.117173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.117148    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2c7ca6\""
Oct 30 21:22:20.121307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.121282    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:22:20.172095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.172055    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:22:20.178650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.178580    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.191381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.191338    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdnm\" (UniqueName: \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.191606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.191390    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.199208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.199178    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2c7ca6-dockercfg-zcxt4\""
Oct 30 21:22:20.200099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.200075    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:22:20.349509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.347580    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.349509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.347654    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdnm\" (UniqueName: \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.349509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.347755    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.349509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.347790    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzntr\" (UniqueName: \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.349880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.349531    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.375550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.372817    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdnm\" (UniqueName: \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\") pod \"payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") " pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.415970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.415931    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:20.449450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.449417    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.449630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.449544    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzntr\" (UniqueName: \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.450248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.450225    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.474460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.474422    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzntr\" (UniqueName: \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\") pod \"payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.499585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.498968    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:20.742767 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:20.742725    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a38aa1_189b_41e0_9388_065575036560.slice/crio-97924ae29cf17da671aad0d23616990e793f00b1fd950577312032d0027571f4 WatchSource:0}: Error finding container 97924ae29cf17da671aad0d23616990e793f00b1fd950577312032d0027571f4: Status 404 returned error can't find the container with id 97924ae29cf17da671aad0d23616990e793f00b1fd950577312032d0027571f4
Oct 30 21:22:20.758288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.758261    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:22:20.916819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.914180    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:22:20.957079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.957046    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" event={"ID":"55bea80c-2b92-4d7f-b11a-803ba7f30978","Type":"ContainerStarted","Data":"f1ece9533ad8ec65ee7384cad23d21149ead6284e0d3e8837bfd44c33975f2cd"}
Oct 30 21:22:20.959725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:20.959695    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" event={"ID":"91a38aa1-189b-41e0-9388-065575036560","Type":"ContainerStarted","Data":"97924ae29cf17da671aad0d23616990e793f00b1fd950577312032d0027571f4"}
Oct 30 21:22:21.969856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:21.969816    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" event={"ID":"55bea80c-2b92-4d7f-b11a-803ba7f30978","Type":"ContainerStarted","Data":"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"}
Oct 30 21:22:21.970324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:21.969889    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:21.971940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:21.971912    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" event={"ID":"91a38aa1-189b-41e0-9388-065575036560","Type":"ContainerStarted","Data":"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"}
Oct 30 21:22:21.972110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:21.971981    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:22.024911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.024858    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" podStartSLOduration=2.024844792 podStartE2EDuration="2.024844792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:22.021543081 +0000 UTC m=+1285.433082515" watchObservedRunningTime="2026-10-30 21:22:22.024844792 +0000 UTC m=+1285.436384228"
Oct 30 21:22:22.065134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.065088    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" podStartSLOduration=2.065074895 podStartE2EDuration="2.065074895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:22.064391652 +0000 UTC m=+1285.475931088" watchObservedRunningTime="2026-10-30 21:22:22.065074895 +0000 UTC m=+1285.476614360"
Oct 30 21:22:22.263878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.263799    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:22:22.264320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.264248    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:22:22.618058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.618028    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:22:22.717992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.717959    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.718339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.718319    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.718755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.718734    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.718948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.718933    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.719070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.719056    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.719207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.719191    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.719346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.719333    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.719838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.719805    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.719968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.719952    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\") pod \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\" (UID: \"cfa94e31-814c-49d4-9cc8-0d5f6e11c620\") "
Oct 30 21:22:22.726535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.726158    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.726535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.726304    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.726535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.726528    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.729738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.729687    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.729994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.729969    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.729994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.729971    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.730162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.730142    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.730290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.730273    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.735391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.735363    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq" pod "cfa94e31-814c-49d4-9cc8-0d5f6e11c620" (UID: "cfa94e31-814c-49d4-9cc8-0d5f6e11c620"). InnerVolumeSpecName "kube-api-access-g4cbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.782260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.782060    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:22:22.788814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.788795    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:22:22.791225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.791194    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:22:22.805596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.805563    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826275    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826361    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826397    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826432    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826531    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826574    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826626    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826656    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826714    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826757    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826777    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826819    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826911    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.826968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826946    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.826987    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827023    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827045    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827056    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827107    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827141    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827235    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827283    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827377    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827466    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827465    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827522    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827557    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827640    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827667    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827696    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827725    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.827749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827743    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827819    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827884    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827947    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.827988    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.828019    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\") pod \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\" (UID: \"10424c1b-c3f5-449b-8369-c6acb60ba8d2\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.828060    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\") pod \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\" (UID: \"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.828120    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\") pod \"0f14a956-ae09-485f-b5e0-8390677d1c92\" (UID: \"0f14a956-ae09-485f-b5e0-8390677d1c92\") "
Oct 30 21:22:22.828814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.828152    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\") pod \"957e73ad-368b-4900-9839-ae9669b6858a\" (UID: \"957e73ad-368b-4900-9839-ae9669b6858a\") "
Oct 30 21:22:22.829241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829217    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829248    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829262    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829276    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829289    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829303    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829316    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829330    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957e73ad-368b-4900-9839-ae9669b6858a-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829345    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829357    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829370    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829383    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cbq\" (UniqueName: \"kubernetes.io/projected/cfa94e31-814c-49d4-9cc8-0d5f6e11c620-kube-api-access-g4cbq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829398    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829324    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.829542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829522    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.830105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.829743    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.830298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.830276    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.830741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.830458    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.831733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.831699    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.831888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.831869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.831989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.831888    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.832401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.832372    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "kube-api-access-254sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.832675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.832647    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.833570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.833222    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.834019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.833929    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.836533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.834532    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "kube-api-access-69s92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.836533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.835210    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.842244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.842210    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.843019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.842823    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.843313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.843146    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.845345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.844533    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "kube-api-access-75kz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.845345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.845174    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.846971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.845442    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.846971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.846598    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.846971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.846774    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.848379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.848343    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy" pod "10424c1b-c3f5-449b-8369-c6acb60ba8d2" (UID: "10424c1b-c3f5-449b-8369-c6acb60ba8d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.848933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.848890    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.849452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.849433    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.850890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.849821    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.851465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.851433    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.852681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.852650    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.852995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.852965    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo" pod "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" (UID: "27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.853450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.853420    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4" pod "0f14a956-ae09-485f-b5e0-8390677d1c92" (UID: "0f14a956-ae09-485f-b5e0-8390677d1c92"). InnerVolumeSpecName "kube-api-access-jmpq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.857860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.857817    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.859394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.859358    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy" pod "957e73ad-368b-4900-9839-ae9669b6858a" (UID: "957e73ad-368b-4900-9839-ae9669b6858a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.895300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.895272    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:22:22.930023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.929989    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930111    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930240    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930300    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.930386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930337    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.930386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930341    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930414    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930452    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930525    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930604    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.930789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930718    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.930950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930923    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.931024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.930986    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:22.931077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931040    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\") pod \"473d9566-a4af-4747-8ca4-25c9b33d7e63\" (UID: \"473d9566-a4af-4747-8ca4-25c9b33d7e63\") "
Oct 30 21:22:22.931887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931862    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931891    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931905    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931919    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931932    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931945    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931957    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931970    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.931993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931983    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.933390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.931996    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.933545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.933529    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.933681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.933649    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.933837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.933821    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.933945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.933933    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934033    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69s92\" (UniqueName: \"kubernetes.io/projected/957e73ad-368b-4900-9839-ae9669b6858a-kube-api-access-69s92\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934057    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473d9566-a4af-4747-8ca4-25c9b33d7e63-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934070    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934082    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934095    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934108    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934121    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254sm\" (UniqueName: \"kubernetes.io/projected/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-kube-api-access-254sm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934135    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934148    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934161    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957e73ad-368b-4900-9839-ae9669b6858a-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934174    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934186    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f14a956-ae09-485f-b5e0-8390677d1c92-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934199    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934212    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934226    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934239    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f14a956-ae09-485f-b5e0-8390677d1c92-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934251    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934265    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpq4\" (UniqueName: \"kubernetes.io/projected/0f14a956-ae09-485f-b5e0-8390677d1c92-kube-api-access-jmpq4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934277    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934290    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957e73ad-368b-4900-9839-ae9669b6858a-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934304    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kz9\" (UniqueName: \"kubernetes.io/projected/10424c1b-c3f5-449b-8369-c6acb60ba8d2-kube-api-access-75kz9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934319    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.934413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934332    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10424c1b-c3f5-449b-8369-c6acb60ba8d2-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:22.935059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934720    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "kube-api-access-vq6cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.935059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.934789    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:22.935232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.935210    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.936373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.936353    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo" pod "473d9566-a4af-4747-8ca4-25c9b33d7e63" (UID: "473d9566-a4af-4747-8ca4-25c9b33d7e63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:22.987887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.987850    7037 generic.go:356] "Generic (PLEG): container finished" podID="957e73ad-368b-4900-9839-ae9669b6858a" containerID="50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8" exitCode=0
Oct 30 21:22:22.988287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.987955    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" event={"ID":"957e73ad-368b-4900-9839-ae9669b6858a","Type":"ContainerDied","Data":"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"}
Oct 30 21:22:22.988287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.988017    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9" event={"ID":"957e73ad-368b-4900-9839-ae9669b6858a","Type":"ContainerDied","Data":"75384eda3204c993c26e6730cdceb206d39bff34b56b3b5a4fa18b735d85e6cd"}
Oct 30 21:22:22.988287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.988046    7037 scope.go:122] "RemoveContainer" containerID="50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"
Oct 30 21:22:22.988287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.988203    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"
Oct 30 21:22:22.998420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.998394    7037 generic.go:356] "Generic (PLEG): container finished" podID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerID="20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0" exitCode=0
Oct 30 21:22:22.998667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.998480    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" event={"ID":"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f","Type":"ContainerDied","Data":"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"}
Oct 30 21:22:22.998918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.998666    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg" event={"ID":"27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f","Type":"ContainerDied","Data":"0af017dfbbba8c6924fb176a7d7d53aedc3284104128a3f315411bfeb7444dca"}
Oct 30 21:22:22.998918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:22.998758    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"
Oct 30 21:22:23.006295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.006268    7037 generic.go:356] "Generic (PLEG): container finished" podID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerID="e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5" exitCode=0
Oct 30 21:22:23.006419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.006329    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"
Oct 30 21:22:23.006419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.006345    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerDied","Data":"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"}
Oct 30 21:22:23.006419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.006401    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn" event={"ID":"10424c1b-c3f5-449b-8369-c6acb60ba8d2","Type":"ContainerDied","Data":"74c8ad245303d19ac7bff314bf5470e15db5d70c12759f155da1f957351cd145"}
Oct 30 21:22:23.008356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.008291    7037 generic.go:356] "Generic (PLEG): container finished" podID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerID="640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a" exitCode=0
Oct 30 21:22:23.008768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.008386    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"
Oct 30 21:22:23.008768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.008440    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" event={"ID":"0f14a956-ae09-485f-b5e0-8390677d1c92","Type":"ContainerDied","Data":"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"}
Oct 30 21:22:23.008768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.008511    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm" event={"ID":"0f14a956-ae09-485f-b5e0-8390677d1c92","Type":"ContainerDied","Data":"5a03a406952a76c36cf9eded29811e65d7398315bc6250a8b8e92996d5ad783f"}
Oct 30 21:22:23.030673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.030633    7037 scope.go:122] "RemoveContainer" containerID="50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"
Oct 30 21:22:23.031675 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.031645    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8\": container with ID starting with 50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8 not found: ID does not exist" containerID="50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"
Oct 30 21:22:23.031866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.031839    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8"} err="failed to get container status \"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8\": rpc error: code = NotFound desc = could not find container \"50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8\": container with ID starting with 50051066854c5c28ba5bf403aa00cec3236981ac46ea32d47420d55f34275fb8 not found: ID does not exist"
Oct 30 21:22:23.031972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.031957    7037 scope.go:122] "RemoveContainer" containerID="20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"
Oct 30 21:22:23.034137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.034114    7037 generic.go:356] "Generic (PLEG): container finished" podID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerID="be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21" exitCode=0
Oct 30 21:22:23.034318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.034303    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"
Oct 30 21:22:23.034642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.034621    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" event={"ID":"473d9566-a4af-4747-8ca4-25c9b33d7e63","Type":"ContainerDied","Data":"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"}
Oct 30 21:22:23.034798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.034780    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw" event={"ID":"473d9566-a4af-4747-8ca4-25c9b33d7e63","Type":"ContainerDied","Data":"834d17c1fee3e7b72ac3f9d22bd3f653b1b4abd29a1cc9a8b6ce39021f829ef6"}
Oct 30 21:22:23.036828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.036806    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:23.036972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.036958    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:23.037078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.037065    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6cf\" (UniqueName: \"kubernetes.io/projected/473d9566-a4af-4747-8ca4-25c9b33d7e63-kube-api-access-vq6cf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:23.037163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.037151    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473d9566-a4af-4747-8ca4-25c9b33d7e63-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:23.056646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.056623    7037 generic.go:356] "Generic (PLEG): container finished" podID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerID="0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e" exitCode=0
Oct 30 21:22:23.056819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.056795    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" event={"ID":"cfa94e31-814c-49d4-9cc8-0d5f6e11c620","Type":"ContainerDied","Data":"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"}
Oct 30 21:22:23.056914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.056860    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk" event={"ID":"cfa94e31-814c-49d4-9cc8-0d5f6e11c620","Type":"ContainerDied","Data":"60af31d05c2f0f964cded63a00632ccd101021674a9ef09edfe3b3b37a122fea"}
Oct 30 21:22:23.056914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.056782    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"
Oct 30 21:22:23.057363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.056677    7037 scope.go:122] "RemoveContainer" containerID="20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"
Oct 30 21:22:23.058764 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.058393    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0\": container with ID starting with 20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0 not found: ID does not exist" containerID="20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"
Oct 30 21:22:23.058764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.058428    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0"} err="failed to get container status \"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0\": rpc error: code = NotFound desc = could not find container \"20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0\": container with ID starting with 20504af60865229d251805526f00c7052430a02e9956a124ad3fa4f616e65ef0 not found: ID does not exist"
Oct 30 21:22:23.058764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.058449    7037 scope.go:122] "RemoveContainer" containerID="e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"
Oct 30 21:22:23.059559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.059535    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:22:23.068410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.068383    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-hq8g9"]
Oct 30 21:22:23.071120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.071007    7037 scope.go:122] "RemoveContainer" containerID="a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"
Oct 30 21:22:23.084226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.084211    7037 scope.go:122] "RemoveContainer" containerID="e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"
Oct 30 21:22:23.084543 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.084467    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5\": container with ID starting with e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5 not found: ID does not exist" containerID="e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"
Oct 30 21:22:23.084543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.084527    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5"} err="failed to get container status \"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5\": rpc error: code = NotFound desc = could not find container \"e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5\": container with ID starting with e979041b09962163a3e43e59af9fed34c480fa8a90aa533fe038b835841000b5 not found: ID does not exist"
Oct 30 21:22:23.084676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.084546    7037 scope.go:122] "RemoveContainer" containerID="a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"
Oct 30 21:22:23.084815 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.084800    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0\": container with ID starting with a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0 not found: ID does not exist" containerID="a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"
Oct 30 21:22:23.084860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.084820    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0"} err="failed to get container status \"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0\": rpc error: code = NotFound desc = could not find container \"a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0\": container with ID starting with a2d0d1c11a51e41a55632bc798c4a96207c13d59f5e9a87f7fc0a4245512fbc0 not found: ID does not exist"
Oct 30 21:22:23.084860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.084835    7037 scope.go:122] "RemoveContainer" containerID="640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"
Oct 30 21:22:23.097068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.096989    7037 scope.go:122] "RemoveContainer" containerID="640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"
Oct 30 21:22:23.097283 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.097258    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a\": container with ID starting with 640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a not found: ID does not exist" containerID="640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"
Oct 30 21:22:23.097372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.097287    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a"} err="failed to get container status \"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a\": rpc error: code = NotFound desc = could not find container \"640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a\": container with ID starting with 640ae66e480a957f1bde3e045363919696b50360eb5a5cd14f38e3b3da4b9e0a not found: ID does not exist"
Oct 30 21:22:23.097372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.097310    7037 scope.go:122] "RemoveContainer" containerID="be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"
Oct 30 21:22:23.099832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.099806    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:22:23.104116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.104095    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-jh5vg"]
Oct 30 21:22:23.107957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.107941    7037 scope.go:122] "RemoveContainer" containerID="be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"
Oct 30 21:22:23.108225 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.108208    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21\": container with ID starting with be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21 not found: ID does not exist" containerID="be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"
Oct 30 21:22:23.108330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.108230    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21"} err="failed to get container status \"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21\": rpc error: code = NotFound desc = could not find container \"be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21\": container with ID starting with be19e2094fc9910627b46e84eb911494bb0494be563e219aeddf87f33e7ecc21 not found: ID does not exist"
Oct 30 21:22:23.108330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.108249    7037 scope.go:122] "RemoveContainer" containerID="0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"
Oct 30 21:22:23.118316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.118288    7037 scope.go:122] "RemoveContainer" containerID="0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"
Oct 30 21:22:23.118604 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:23.118586    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e\": container with ID starting with 0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e not found: ID does not exist" containerID="0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"
Oct 30 21:22:23.118688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.118609    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e"} err="failed to get container status \"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e\": rpc error: code = NotFound desc = could not find container \"0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e\": container with ID starting with 0d0ea23ec79c7b0209f7390fc676a2c42f586a77ecfe011afb7cdcd0d3bc2b8e not found: ID does not exist"
Oct 30 21:22:23.172010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.171973    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:22:23.176024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.175998    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-n6gtk"]
Oct 30 21:22:23.207027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.206999    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:22:23.213450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.213427    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-g9vrm"]
Oct 30 21:22:23.254547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.254518    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:22:23.257951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.257930    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-2tbbn"]
Oct 30 21:22:23.291817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.291780    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:22:23.297126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.297103    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-d8rvw"]
Oct 30 21:22:23.743557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.743523    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744656    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744683    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744731    7037 cpu_manager.go:414] "containerMap: removing container" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744740    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744760    7037 cpu_manager.go:414] "containerMap: removing container" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744768    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744808    7037 cpu_manager.go:414] "containerMap: removing container" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744816    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744830    7037 cpu_manager.go:414] "containerMap: removing container" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744838    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744878    7037 cpu_manager.go:414] "containerMap: removing container" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy"
Oct 30 21:22:23.744993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.744886    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745035    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745047    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745058    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745065    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745076    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="957e73ad-368b-4900-9839-ae9669b6858a" containerName="istio-proxy"
Oct 30 21:22:23.745835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.745086    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" containerName="istio-proxy"
Oct 30 21:22:23.751414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.751395    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.771233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.771192    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-908469\""
Oct 30 21:22:23.771406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.771391    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-908469-dockercfg-cxh8f\""
Oct 30 21:22:23.776483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.776452    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:22:23.834059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.834026    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:22:23.835355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.835330    7037 cpu_manager.go:414] "containerMap: removing container" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.835481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.835355    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.835611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.835596    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" containerName="istio-proxy"
Oct 30 21:22:23.840564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.840540    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:23.859985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.859951    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.860108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.860053    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82p5j\" (UniqueName: \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.866641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.866606    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:22:23.961551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.961511    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.961883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.961854    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82p5j\" (UniqueName: \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.962011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.961962    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:23.962400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.962366    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:23.963083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.962078    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cprn\" (UniqueName: \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:23.981570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:23.981549    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82p5j\" (UniqueName: \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\") pod \"payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:24.064662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.064577    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:24.065042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.064711    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cprn\" (UniqueName: \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:24.066276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.066249    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:24.083149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.083126    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:24.085459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.085435    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cprn\" (UniqueName: \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\") pod \"payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:24.161997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.161958    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:24.595065 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:24.595028    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248fa86b_6260_4f2a_87b8_12110595f28b.slice/crio-664076d40252cca311e7e7928c0abc9bb77ecff4dcf9230b13fd64f9d0760ba0 WatchSource:0}: Error finding container 664076d40252cca311e7e7928c0abc9bb77ecff4dcf9230b13fd64f9d0760ba0: Status 404 returned error can't find the container with id 664076d40252cca311e7e7928c0abc9bb77ecff4dcf9230b13fd64f9d0760ba0
Oct 30 21:22:24.596212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.596183    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:22:24.704214 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:24.704189    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785797e8_bb2d_4ed3_86ed_68a801f1a38f.slice/crio-a98d0d2d22960d1b64bf87a893ae822a3be8372b1dee298a7a8f10433d085351 WatchSource:0}: Error finding container a98d0d2d22960d1b64bf87a893ae822a3be8372b1dee298a7a8f10433d085351: Status 404 returned error can't find the container with id a98d0d2d22960d1b64bf87a893ae822a3be8372b1dee298a7a8f10433d085351
Oct 30 21:22:24.713692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.713662    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f14a956-ae09-485f-b5e0-8390677d1c92" path="/var/lib/kubelet/pods/0f14a956-ae09-485f-b5e0-8390677d1c92/volumes"
Oct 30 21:22:24.714379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.714354    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10424c1b-c3f5-449b-8369-c6acb60ba8d2" path="/var/lib/kubelet/pods/10424c1b-c3f5-449b-8369-c6acb60ba8d2/volumes"
Oct 30 21:22:24.715037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.715018    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f" path="/var/lib/kubelet/pods/27d64bcd-2ca7-4936-a7a4-bf83c3cebb6f/volumes"
Oct 30 21:22:24.716108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.716088    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473d9566-a4af-4747-8ca4-25c9b33d7e63" path="/var/lib/kubelet/pods/473d9566-a4af-4747-8ca4-25c9b33d7e63/volumes"
Oct 30 21:22:24.716745 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.716726    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957e73ad-368b-4900-9839-ae9669b6858a" path="/var/lib/kubelet/pods/957e73ad-368b-4900-9839-ae9669b6858a/volumes"
Oct 30 21:22:24.717335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.717316    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa94e31-814c-49d4-9cc8-0d5f6e11c620" path="/var/lib/kubelet/pods/cfa94e31-814c-49d4-9cc8-0d5f6e11c620/volumes"
Oct 30 21:22:24.718154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:24.718137    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:22:25.093134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.093094    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" event={"ID":"785797e8-bb2d-4ed3-86ed-68a801f1a38f","Type":"ContainerStarted","Data":"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"}
Oct 30 21:22:25.093631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.093165    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:25.093631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.093190    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" event={"ID":"785797e8-bb2d-4ed3-86ed-68a801f1a38f","Type":"ContainerStarted","Data":"a98d0d2d22960d1b64bf87a893ae822a3be8372b1dee298a7a8f10433d085351"}
Oct 30 21:22:25.095052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.095025    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" event={"ID":"248fa86b-6260-4f2a-87b8-12110595f28b","Type":"ContainerStarted","Data":"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"}
Oct 30 21:22:25.095168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.095071    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" event={"ID":"248fa86b-6260-4f2a-87b8-12110595f28b","Type":"ContainerStarted","Data":"664076d40252cca311e7e7928c0abc9bb77ecff4dcf9230b13fd64f9d0760ba0"}
Oct 30 21:22:25.095168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.095112    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:25.139579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.139527    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" podStartSLOduration=2.139514179 podStartE2EDuration="2.139514179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:25.137315279 +0000 UTC m=+1288.548854714" watchObservedRunningTime="2026-10-30 21:22:25.139514179 +0000 UTC m=+1288.551053614"
Oct 30 21:22:25.203204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.203152    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" podStartSLOduration=2.20313689 podStartE2EDuration="2.20313689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:25.200067312 +0000 UTC m=+1288.611606748" watchObservedRunningTime="2026-10-30 21:22:25.20313689 +0000 UTC m=+1288.614676325"
Oct 30 21:22:25.264664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.264626    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:22:25.271040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.271015    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.301599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.301479    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:22:25.354934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.354836    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-59fb17-openshift-default-dockercfg-v8czt\""
Oct 30 21:22:25.413737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.413703    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.413914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.413870    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.413971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.413916    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnk7\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.413971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.413949    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.414055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.414033    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.414103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.414090    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.414218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.414197    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.414260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.414232    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.414412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.414391    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.515845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.515806    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.515941    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.515975    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnk7\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516020    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516051    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516118    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516183    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516203    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516219    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516369    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516535    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516564    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516760 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516733    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.516958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.516940    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.518335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.518314    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.518579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.518560    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.541694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.541665    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.542022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.542003    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnk7\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:25.593572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:25.593528    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:26.143376 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:26.143339    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723ff275_9ef2_426a_979e_04728732bb1e.slice/crio-49d219dc03b61dc02122816bc3f2be15cdb82bee40b1f44ee68e1d1e82ca58c0 WatchSource:0}: Error finding container 49d219dc03b61dc02122816bc3f2be15cdb82bee40b1f44ee68e1d1e82ca58c0: Status 404 returned error can't find the container with id 49d219dc03b61dc02122816bc3f2be15cdb82bee40b1f44ee68e1d1e82ca58c0
Oct 30 21:22:26.143806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:26.143736    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:22:26.146655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:26.146617    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:22:26.146743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:26.146688    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:22:26.146779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:26.146719    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:22:27.158309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.158265    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" event={"ID":"723ff275-9ef2-426a-979e-04728732bb1e","Type":"ContainerStarted","Data":"cf6cd9dcab388aa47c67dcdc417a9976bbd1f6638afa3d000be01af3ad1a98e6"}
Oct 30 21:22:27.158917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.158890    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" event={"ID":"723ff275-9ef2-426a-979e-04728732bb1e","Type":"ContainerStarted","Data":"49d219dc03b61dc02122816bc3f2be15cdb82bee40b1f44ee68e1d1e82ca58c0"}
Oct 30 21:22:27.214621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.214437    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podStartSLOduration=2.214418479 podStartE2EDuration="2.214418479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:27.208409362 +0000 UTC m=+1290.619948809" watchObservedRunningTime="2026-10-30 21:22:27.214418479 +0000 UTC m=+1290.625957918"
Oct 30 21:22:27.594288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.594251    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:27.596173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.596147    7037 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 21:22:27.596259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:27.596220    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 21:22:28.594568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:28.594529    7037 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 21:22:28.594992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:28.594586    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 21:22:29.594253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:29.594217    7037 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 21:22:29.594429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:29.594273    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 21:22:30.594628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:30.594597    7037 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 21:22:30.595001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:30.594652    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 21:22:30.963894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:30.963804    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:22:31.594210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:31.594167    7037 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 21:22:31.594387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:31.594239    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 21:22:31.975459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:31.975382    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:22:32.598886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:32.598843    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:32.599327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:32.599304    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:32.600023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:32.600001    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:22:33.124012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:33.123682    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:22:33.134772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:33.134735    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:22:33.674396 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:33.674349    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:22:36.109265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:36.109236    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:22:36.109790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:36.109303    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:22:39.907072 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.907032    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:22:39.912048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.912024    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:22:39.912171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.912140    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:39.916794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.916775    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:39.932882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.932853    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:22:39.937283 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.937253    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:22:39.939640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.939606    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ca18b3-dockercfg-cjjgt\""
Oct 30 21:22:39.939640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:39.939620    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ca18b3\""
Oct 30 21:22:40.042226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.042186    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.042438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.042246    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjvr\" (UniqueName: \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.042623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.042594    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njp5q\" (UniqueName: \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.042698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.042652    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.146724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.146684    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.146962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.146950    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjvr\" (UniqueName: \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.147163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.147153    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp5q\" (UniqueName: \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.147246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.147237    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.148039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.148018    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.148683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.148666    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.177342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.177263    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp5q\" (UniqueName: \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\") pod \"payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.180882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.180851    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjvr\" (UniqueName: \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\") pod \"payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.231747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.231711    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:40.246434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.246400    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:40.552357 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:40.552313    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7420e6dd_23a0_4fe5_b6d8_58a4d8b22f7d.slice/crio-ff4faf56be7f8c4edfec407bf35f9d59b8be3df7ae7069a028de004da9881d88 WatchSource:0}: Error finding container ff4faf56be7f8c4edfec407bf35f9d59b8be3df7ae7069a028de004da9881d88: Status 404 returned error can't find the container with id ff4faf56be7f8c4edfec407bf35f9d59b8be3df7ae7069a028de004da9881d88
Oct 30 21:22:40.553873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.553849    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:22:40.575361 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:40.575327    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f9dde6_6e58_4656_a033_356592b117fc.slice/crio-3558c8fd366d95a2041b243295f02028db15516d7d638d8e4210016a57e735e7 WatchSource:0}: Error finding container 3558c8fd366d95a2041b243295f02028db15516d7d638d8e4210016a57e735e7: Status 404 returned error can't find the container with id 3558c8fd366d95a2041b243295f02028db15516d7d638d8e4210016a57e735e7
Oct 30 21:22:40.578972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:40.578950    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:22:41.331184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.331147    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" event={"ID":"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d","Type":"ContainerStarted","Data":"c9fd386a3c3c206ad41592a3918434b9793d6e36839f7c690765583edf5b67c7"}
Oct 30 21:22:41.331696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.331352    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:41.331696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.331406    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" event={"ID":"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d","Type":"ContainerStarted","Data":"ff4faf56be7f8c4edfec407bf35f9d59b8be3df7ae7069a028de004da9881d88"}
Oct 30 21:22:41.333380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.333349    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" event={"ID":"30f9dde6-6e58-4656-a033-356592b117fc","Type":"ContainerStarted","Data":"aea6ac697a4e211ca368c7e109b7e1644c1d24a67381857a7684195abdfee5fc"}
Oct 30 21:22:41.333530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.333404    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" event={"ID":"30f9dde6-6e58-4656-a033-356592b117fc","Type":"ContainerStarted","Data":"3558c8fd366d95a2041b243295f02028db15516d7d638d8e4210016a57e735e7"}
Oct 30 21:22:41.333530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.333429    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:41.372455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.372400    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" podStartSLOduration=2.372385979 podStartE2EDuration="2.372385979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:41.369340531 +0000 UTC m=+1304.780879977" watchObservedRunningTime="2026-10-30 21:22:41.372385979 +0000 UTC m=+1304.783925414"
Oct 30 21:22:41.409563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:41.409512    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" podStartSLOduration=2.409480066 podStartE2EDuration="2.409480066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:41.408640038 +0000 UTC m=+1304.820179473" watchObservedRunningTime="2026-10-30 21:22:41.409480066 +0000 UTC m=+1304.821019502"
Oct 30 21:22:45.698897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:45.698859    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:22:45.699342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:45.699079    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy" containerID="cri-o://1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8" gracePeriod=30
Oct 30 21:22:47.400502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.400453    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:22:47.416003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.415970    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.438124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.438089    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:22:47.443537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.443506    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4266b7-serving-cert\""
Oct 30 21:22:47.506360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.506322    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.506649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.506374    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.506649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.506551    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s5k\" (UniqueName: \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.506799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.506672    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.608297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.608258    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.608527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.608313    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.608527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.608404    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s5k\" (UniqueName: \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.608527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.608505    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.609232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.609205    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.610952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.610920    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.611174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.611154    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.651479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.640591    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s5k\" (UniqueName: \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\") pod \"maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:47.760692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:47.760651    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:48.098954 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:48.098925    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ca7c2a_c7dc_4fa0_a1cc_094643011b26.slice/crio-cb736c02cacef440ad0502fb3e1448795b70ff3e485054d6b0ff48b815a18445 WatchSource:0}: Error finding container cb736c02cacef440ad0502fb3e1448795b70ff3e485054d6b0ff48b815a18445: Status 404 returned error can't find the container with id cb736c02cacef440ad0502fb3e1448795b70ff3e485054d6b0ff48b815a18445
Oct 30 21:22:48.103418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:48.103392    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:22:48.399727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:48.399641    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" event={"ID":"92ca7c2a-c7dc-4fa0-a1cc-094643011b26","Type":"ContainerStarted","Data":"cb736c02cacef440ad0502fb3e1448795b70ff3e485054d6b0ff48b815a18445"}
Oct 30 21:22:49.194409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.194375    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:22:49.199159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.199136    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.218286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.218254    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2c7ca6-serving-cert\""
Oct 30 21:22:49.227484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.227449    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:22:49.248528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.242877    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.248528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.243033    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.248528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.243252    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.248528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.243309    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxpk\" (UniqueName: \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.344468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.344437    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.344656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.344524    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxpk\" (UniqueName: \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.344656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.344624    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.344761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.344746    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.344917 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:49.344894    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-serving-cert: secret "maas-api-e2e-api-b-2c7ca6-serving-cert" not found
Oct 30 21:22:49.344986 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:49.344974    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls podName:f9b50a7a-ad6d-461c-894b-5c1c3ef3954a nodeName:}" failed. No retries permitted until 2026-10-30 21:22:49.84495343 +0000 UTC m=+1313.256492856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls") pod "maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" (UID: "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a") : secret "maas-api-e2e-api-b-2c7ca6-serving-cert" not found
Oct 30 21:22:49.345211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.345184    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.347069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.347053    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.374456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.374384    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxpk\" (UniqueName: \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.412040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.412004    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" event={"ID":"92ca7c2a-c7dc-4fa0-a1cc-094643011b26","Type":"ContainerStarted","Data":"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"}
Oct 30 21:22:49.412239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.412129    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:49.474768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.474630    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" podStartSLOduration=2.072951924 podStartE2EDuration="2.474609169s" totalImagesPullingTime="401.657245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:49.467219787 +0000 UTC m=+1312.878759236" watchObservedRunningTime="2026-10-30 21:22:49.474609169 +0000 UTC m=+1312.886148606"
Oct 30 21:22:49.855036 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.855005    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:49.857363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:49.857341    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") pod \"maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:50.120414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:50.120335    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:50.441265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:50.441232    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:22:50.448843 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:50.448800    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b50a7a_ad6d_461c_894b_5c1c3ef3954a.slice/crio-5cee4329bab81a166926e69319b985971a5d693293b6b136dd4e3f0b36a6c7b2 WatchSource:0}: Error finding container 5cee4329bab81a166926e69319b985971a5d693293b6b136dd4e3f0b36a6c7b2: Status 404 returned error can't find the container with id 5cee4329bab81a166926e69319b985971a5d693293b6b136dd4e3f0b36a6c7b2
Oct 30 21:22:51.009216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.009188    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:22:51.082660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.082625    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.082813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.082730    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.082813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.082794    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.082995    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083079    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083092    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.083218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083154    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083192    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:51.083218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083207    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083263    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083324    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\") pod \"df432a47-d0e6-474d-91b4-78c1b5518385\" (UID: \"df432a47-d0e6-474d-91b4-78c1b5518385\") "
Oct 30 21:22:51.083371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083357    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.083564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083522    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.083564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.083523    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.084313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.084286    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.084440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.084316    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.084440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.084333    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df432a47-d0e6-474d-91b4-78c1b5518385-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.084440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.084346    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.084440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.084360    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.085601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.085568    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:51.085859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.085835    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.085947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.085860    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "kube-api-access-kmn26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:51.085947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.085887    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token" pod "df432a47-d0e6-474d-91b4-78c1b5518385" (UID: "df432a47-d0e6-474d-91b4-78c1b5518385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:51.185763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.185727    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.185763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.185757    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmn26\" (UniqueName: \"kubernetes.io/projected/df432a47-d0e6-474d-91b4-78c1b5518385-kube-api-access-kmn26\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.185763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.185774    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df432a47-d0e6-474d-91b4-78c1b5518385-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.186069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.185783    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df432a47-d0e6-474d-91b4-78c1b5518385-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:22:51.433800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.433762    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" event={"ID":"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a","Type":"ContainerStarted","Data":"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"}
Oct 30 21:22:51.434001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.433830    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" event={"ID":"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a","Type":"ContainerStarted","Data":"5cee4329bab81a166926e69319b985971a5d693293b6b136dd4e3f0b36a6c7b2"}
Oct 30 21:22:51.434001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.433866    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:51.435505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.435471    7037 generic.go:356] "Generic (PLEG): container finished" podID="df432a47-d0e6-474d-91b4-78c1b5518385" containerID="1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8" exitCode=0
Oct 30 21:22:51.435614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.435518    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"
Oct 30 21:22:51.435614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.435560    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" event={"ID":"df432a47-d0e6-474d-91b4-78c1b5518385","Type":"ContainerDied","Data":"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"}
Oct 30 21:22:51.435614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.435604    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp" event={"ID":"df432a47-d0e6-474d-91b4-78c1b5518385","Type":"ContainerDied","Data":"3c49d4f48da33ccdbb4439f3a8c7258d6f93b46e325c5d8d9600426ec58d382a"}
Oct 30 21:22:51.435739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.435629    7037 scope.go:122] "RemoveContainer" containerID="1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"
Oct 30 21:22:51.452588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.452569    7037 scope.go:122] "RemoveContainer" containerID="1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"
Oct 30 21:22:51.452945 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:22:51.452925    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8\": container with ID starting with 1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8 not found: ID does not exist" containerID="1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"
Oct 30 21:22:51.453012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.452954    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8"} err="failed to get container status \"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8\": rpc error: code = NotFound desc = could not find container \"1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8\": container with ID starting with 1adc3ea0eea9c47697612cb6adbea59ae051e0ffa1b0dddd32c47844d9019cf8 not found: ID does not exist"
Oct 30 21:22:51.480526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.480459    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" podStartSLOduration=2.172862883 podStartE2EDuration="2.480446721s" totalImagesPullingTime="307.583838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:51.476201215 +0000 UTC m=+1314.887740644" watchObservedRunningTime="2026-10-30 21:22:51.480446721 +0000 UTC m=+1314.891986156"
Oct 30 21:22:51.519892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.519866    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:22:51.524421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:51.524396    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-pw8sp"]
Oct 30 21:22:52.344798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.344768    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:22:52.344988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.344848    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:22:52.711105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.711032    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" path="/var/lib/kubelet/pods/df432a47-d0e6-474d-91b4-78c1b5518385/volumes"
Oct 30 21:22:52.800374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.800334    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:22:52.801783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.801759    7037 cpu_manager.go:414] "containerMap: removing container" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy"
Oct 30 21:22:52.801838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.801786    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy"
Oct 30 21:22:52.802067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.802050    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="df432a47-d0e6-474d-91b4-78c1b5518385" containerName="istio-proxy"
Oct 30 21:22:52.808362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.808337    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:52.842480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.842450    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-908469-serving-cert\""
Oct 30 21:22:52.846325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.846295    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:22:52.916586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.916547    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:52.916793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.916772    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:52.916864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.916827    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7wj\" (UniqueName: \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:52.916864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:52.916846    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.018435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.018352    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.018435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.018428    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7wj\" (UniqueName: \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.018731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.018454    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.018731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.018683    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.019339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.019315    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.021145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.021120    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.021248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.021120    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.041602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.038525    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7wj\" (UniqueName: \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\") pod \"maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.145551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.142955    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:53.470973 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:22:53.470942    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f1d12_9691_4eed_ac7d_2b34dab74c27.slice/crio-ecc6b8c788873e65ecf45b364c66e58b0ab1e82769cf69da357541254a073f76 WatchSource:0}: Error finding container ecc6b8c788873e65ecf45b364c66e58b0ab1e82769cf69da357541254a073f76: Status 404 returned error can't find the container with id ecc6b8c788873e65ecf45b364c66e58b0ab1e82769cf69da357541254a073f76
Oct 30 21:22:53.472882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.472853    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:22:53.578300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:53.569107    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" event={"ID":"708f1d12-9691-4eed-ac7d-2b34dab74c27","Type":"ContainerStarted","Data":"ecc6b8c788873e65ecf45b364c66e58b0ab1e82769cf69da357541254a073f76"}
Oct 30 21:22:54.582375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:54.582334    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" event={"ID":"708f1d12-9691-4eed-ac7d-2b34dab74c27","Type":"ContainerStarted","Data":"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"}
Oct 30 21:22:54.582901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:54.582671    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:22:54.636812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:54.636743    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" podStartSLOduration=2.275911488 podStartE2EDuration="2.636723274s" totalImagesPullingTime="360.811786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:54.628525742 +0000 UTC m=+1318.040065179" watchObservedRunningTime="2026-10-30 21:22:54.636723274 +0000 UTC m=+1318.048262710"
Oct 30 21:22:55.468857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:55.468824    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:22:56.205513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:56.193648    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:56.236514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:56.230881    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:56.265677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:56.265610    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:56.287518 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:56.276733    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:56.997540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:56.992973    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.012276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.012226    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.094687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.094628    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:57.182348 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.182289    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.226857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.225389    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.226857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.225722    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.301794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.277919    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:22:57.377123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.377092    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:22:57.377695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.377659    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:22:57.397583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.397556    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:22:57.407858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.407833    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:22:57.511155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.511125    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:22:57.598708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.598659    7037 scope.go:122] "RemoveContainer" containerID="bd4ec4b585b1d24349ce039d491cbd9e61fd1ecc52760c2583d1cff3ef256241"
Oct 30 21:22:57.621043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.621013    7037 scope.go:122] "RemoveContainer" containerID="f094e89132ef908dbb7fba8862966ba19c13ac257def353391d1190f576bcfcc"
Oct 30 21:22:57.634904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:22:57.634883    7037 scope.go:122] "RemoveContainer" containerID="c8f192d58481558bac46214cf6b9a4107160c1aad9e40da75546980fdaa5db10"
Oct 30 21:23:00.754971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:00.754936    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:23:00.755392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:00.755115    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy" containerID="cri-o://e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4" gracePeriod=30
Oct 30 21:23:01.240955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:01.240921    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:23:02.150893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.150835    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:23:02.151596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.151096    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy" containerID="cri-o://37f5306ebf1c2e62e0dadf930aef20480efd4d0cb8def7490759febed7c2cf5e" gracePeriod=30
Oct 30 21:23:02.187280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.187243    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:23:02.187646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.187608    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy" containerID="cri-o://361c6defbcc6deec49bb18dcc7b8d752e530a8dd7136ef9769b45cc7ec887ee0" gracePeriod=30
Oct 30 21:23:02.887794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.887758    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:23:02.887988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:02.887964    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy" containerID="cri-o://da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6" gracePeriod=30
Oct 30 21:23:03.687584 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:03.687532    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:23:06.075592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.075565    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:23:06.166276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166236    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166301    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166419    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166562    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166596    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166638    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166699    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166818    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:06.166937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166914    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.166994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.166980    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\") pod \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\" (UID: \"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2\") "
Oct 30 21:23:06.167628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.167607    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.169160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169085    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:06.169160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169107    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:06.169160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169137    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "kube-api-access-xwftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:06.169615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169585    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:06.169765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169739    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:06.169833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169742    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:06.169943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.169920    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:06.171307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.171279    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token" pod "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" (UID: "a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:06.268803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268771    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.268803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268799    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.268803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268807    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.269035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268816    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwftc\" (UniqueName: \"kubernetes.io/projected/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-kube-api-access-xwftc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.269035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268824    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.269035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268833    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.269035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268841    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.269035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.268848    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:06.428967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.428935    7037 generic.go:356] "Generic (PLEG): container finished" podID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerID="e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4" exitCode=0
Oct 30 21:23:06.429195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.428996    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" event={"ID":"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2","Type":"ContainerDied","Data":"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"}
Oct 30 21:23:06.429195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.429001    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"
Oct 30 21:23:06.429195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.429069    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598" event={"ID":"a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2","Type":"ContainerDied","Data":"3ac3abe4796b5c8df79c61649d4e29221f1d58f1dd861c933c73f36a673b6154"}
Oct 30 21:23:06.429195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.429100    7037 scope.go:122] "RemoveContainer" containerID="e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"
Oct 30 21:23:06.441060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.441038    7037 scope.go:122] "RemoveContainer" containerID="e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"
Oct 30 21:23:06.441348 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:06.441327    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4\": container with ID starting with e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4 not found: ID does not exist" containerID="e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"
Oct 30 21:23:06.441397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.441356    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4"} err="failed to get container status \"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4\": rpc error: code = NotFound desc = could not find container \"e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4\": container with ID starting with e985750ee9f8cfd85060e7508a0d7a8d18916d8a2257431d9c032285a25d97f4 not found: ID does not exist"
Oct 30 21:23:06.502608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.502573    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:23:06.514434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.514407    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-b9598"]
Oct 30 21:23:06.710201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:06.710126    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" path="/var/lib/kubelet/pods/a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2/volumes"
Oct 30 21:23:07.440609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.440524    7037 generic.go:356] "Generic (PLEG): container finished" podID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerID="361c6defbcc6deec49bb18dcc7b8d752e530a8dd7136ef9769b45cc7ec887ee0" exitCode=0
Oct 30 21:23:07.441027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.440607    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" event={"ID":"9156b2d8-9d48-4320-817e-29fa49050c5d","Type":"ContainerDied","Data":"361c6defbcc6deec49bb18dcc7b8d752e530a8dd7136ef9769b45cc7ec887ee0"}
Oct 30 21:23:07.442321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.442299    7037 generic.go:356] "Generic (PLEG): container finished" podID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerID="37f5306ebf1c2e62e0dadf930aef20480efd4d0cb8def7490759febed7c2cf5e" exitCode=0
Oct 30 21:23:07.442448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.442382    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" event={"ID":"04fd1d12-f119-4cce-9e58-53222ca65c74","Type":"ContainerDied","Data":"37f5306ebf1c2e62e0dadf930aef20480efd4d0cb8def7490759febed7c2cf5e"}
Oct 30 21:23:07.483563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.483543    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:23:07.591080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591043    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591128    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591203    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591352    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591511    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.591640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591574    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591686    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591731    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591776    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591891    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\") pod \"9156b2d8-9d48-4320-817e-29fa49050c5d\" (UID: \"9156b2d8-9d48-4320-817e-29fa49050c5d\") "
Oct 30 21:23:07.591983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.591939    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.592096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592058    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.592559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592294    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.592559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592532    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:07.592730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592719    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.592789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592737    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156b2d8-9d48-4320-817e-29fa49050c5d-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.592789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592750    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.592789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592763    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.592789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.592776    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.593834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.593809    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:07.594336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.594297    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "kube-api-access-bgmg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:07.594432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.594379    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.595133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.595113    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token" pod "9156b2d8-9d48-4320-817e-29fa49050c5d" (UID: "9156b2d8-9d48-4320-817e-29fa49050c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:07.694382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.694310    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.694382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.694343    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.694382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.694356    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmg7\" (UniqueName: \"kubernetes.io/projected/9156b2d8-9d48-4320-817e-29fa49050c5d-kube-api-access-bgmg7\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.694382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.694368    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156b2d8-9d48-4320-817e-29fa49050c5d-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.833523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.833476    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:23:07.896823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.896784    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.897010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.896895    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.897191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.897157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.897313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.897284    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.897889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.897860    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:07.898178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898156    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.898361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898338    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.898613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898575    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.898734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898645    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.898734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898716    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.898898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898807    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.898898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898865    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.898995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.898911    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\") pod \"04fd1d12-f119-4cce-9e58-53222ca65c74\" (UID: \"04fd1d12-f119-4cce-9e58-53222ca65c74\") "
Oct 30 21:23:07.899738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.899712    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.899844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.899754    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.899844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.899769    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.899844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.899782    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04fd1d12-f119-4cce-9e58-53222ca65c74-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.900315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.900290    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.902446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.902422    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:07.902979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.902950    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:07.903276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.903247    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:07.903806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:07.903779    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl" pod "04fd1d12-f119-4cce-9e58-53222ca65c74" (UID: "04fd1d12-f119-4cce-9e58-53222ca65c74"). InnerVolumeSpecName "kube-api-access-rc6cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:08.001468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.001403    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6cl\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-kube-api-access-rc6cl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.001468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.001430    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.001468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.001439    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.001468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.001449    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.001468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.001458    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04fd1d12-f119-4cce-9e58-53222ca65c74-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.223632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.223609    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:23:08.314724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314652    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314714    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314771    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314859    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314876    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314912    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.314965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.314969    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.315235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315004    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.315235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315021    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:08.315235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315075    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:08.315389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315259    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\") pod \"be596f6c-0bd4-4ae7-9075-1040c449f54d\" (UID: \"be596f6c-0bd4-4ae7-9075-1040c449f54d\") "
Oct 30 21:23:08.315531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315258    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:08.315531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315427    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:08.315531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.315447    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:08.316117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.316095    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be596f6c-0bd4-4ae7-9075-1040c449f54d-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.316117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.316117    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.316263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.316126    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.316263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.316134    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.316263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.316143    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.317268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.317243    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:08.317405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.317385    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:08.317447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.317421    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "kube-api-access-599tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:08.317872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.317855    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo" pod "be596f6c-0bd4-4ae7-9075-1040c449f54d" (UID: "be596f6c-0bd4-4ae7-9075-1040c449f54d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:08.417692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.417660    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599tz\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-kube-api-access-599tz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.417692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.417692    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.417899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.417707    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.417899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.417720    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be596f6c-0bd4-4ae7-9075-1040c449f54d-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.459704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.459672    7037 generic.go:356] "Generic (PLEG): container finished" podID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerID="da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6" exitCode=0
Oct 30 21:23:08.460203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.459744    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"
Oct 30 21:23:08.460203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.459745    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" event={"ID":"be596f6c-0bd4-4ae7-9075-1040c449f54d","Type":"ContainerDied","Data":"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"}
Oct 30 21:23:08.460203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.459930    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb" event={"ID":"be596f6c-0bd4-4ae7-9075-1040c449f54d","Type":"ContainerDied","Data":"45f6042ad681c2fb3036e30418b26b066712dc21ab241b67f81359d06812937a"}
Oct 30 21:23:08.460203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.459967    7037 scope.go:122] "RemoveContainer" containerID="da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"
Oct 30 21:23:08.462763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.462741    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"
Oct 30 21:23:08.462840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.462760    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp" event={"ID":"9156b2d8-9d48-4320-817e-29fa49050c5d","Type":"ContainerDied","Data":"fddf1962034cf3b9ff73ccf6a43984f1cdd95395d757aeb73c96a8ffc8ebd07a"}
Oct 30 21:23:08.465207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.465184    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"
Oct 30 21:23:08.465342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.465197    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85" event={"ID":"04fd1d12-f119-4cce-9e58-53222ca65c74","Type":"ContainerDied","Data":"c8f6ae7f9dd987b50436387a92fc71dd178eb7f4b667d5a0053f2f1bd8ec5d62"}
Oct 30 21:23:08.483899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.483881    7037 scope.go:122] "RemoveContainer" containerID="da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"
Oct 30 21:23:08.485976 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:08.485948    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6\": container with ID starting with da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6 not found: ID does not exist" containerID="da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"
Oct 30 21:23:08.486096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.485981    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6"} err="failed to get container status \"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6\": rpc error: code = NotFound desc = could not find container \"da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6\": container with ID starting with da98ee7036dae396b110478e799ccd477d8f019ecb1b6268c55e43aac267e3d6 not found: ID does not exist"
Oct 30 21:23:08.486096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.485998    7037 scope.go:122] "RemoveContainer" containerID="361c6defbcc6deec49bb18dcc7b8d752e530a8dd7136ef9769b45cc7ec887ee0"
Oct 30 21:23:08.497930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.497905    7037 scope.go:122] "RemoveContainer" containerID="37f5306ebf1c2e62e0dadf930aef20480efd4d0cb8def7490759febed7c2cf5e"
Oct 30 21:23:08.519775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.519750    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:23:08.527136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.527112    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-72ptp"]
Oct 30 21:23:08.560282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.560254    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:23:08.564196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.564169    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-5pldb"]
Oct 30 21:23:08.615689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.615654    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:23:08.620406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.620383    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-5th85"]
Oct 30 21:23:08.711712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.711676    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" path="/var/lib/kubelet/pods/04fd1d12-f119-4cce-9e58-53222ca65c74/volumes"
Oct 30 21:23:08.712312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.712291    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" path="/var/lib/kubelet/pods/9156b2d8-9d48-4320-817e-29fa49050c5d/volumes"
Oct 30 21:23:08.712798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:08.712781    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" path="/var/lib/kubelet/pods/be596f6c-0bd4-4ae7-9075-1040c449f54d/volumes"
Oct 30 21:23:30.169943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.169907    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:23:30.170474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.170151    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" podUID="a1807b61-e196-46de-b287-0fc7ce0f42bb" containerName="maas-api" containerID="cri-o://f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6" gracePeriod=30
Oct 30 21:23:30.604583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.604550    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:23:30.648179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.648144    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\") pod \"a1807b61-e196-46de-b287-0fc7ce0f42bb\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") "
Oct 30 21:23:30.648346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.648228    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\") pod \"a1807b61-e196-46de-b287-0fc7ce0f42bb\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") "
Oct 30 21:23:30.648438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.648420    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") pod \"a1807b61-e196-46de-b287-0fc7ce0f42bb\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") "
Oct 30 21:23:30.648507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.648477    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\" (UniqueName: \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\") pod \"a1807b61-e196-46de-b287-0fc7ce0f42bb\" (UID: \"a1807b61-e196-46de-b287-0fc7ce0f42bb\") "
Oct 30 21:23:30.648854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.648824    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca" pod "a1807b61-e196-46de-b287-0fc7ce0f42bb" (UID: "a1807b61-e196-46de-b287-0fc7ce0f42bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:30.649261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.649239    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1807b61-e196-46de-b287-0fc7ce0f42bb-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:30.652082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.652058    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls" pod "a1807b61-e196-46de-b287-0fc7ce0f42bb" (UID: "a1807b61-e196-46de-b287-0fc7ce0f42bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:30.652262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.652236    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs" pod "a1807b61-e196-46de-b287-0fc7ce0f42bb" (UID: "a1807b61-e196-46de-b287-0fc7ce0f42bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:30.652414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.652387    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh" pod "a1807b61-e196-46de-b287-0fc7ce0f42bb" (UID: "a1807b61-e196-46de-b287-0fc7ce0f42bb"). InnerVolumeSpecName "kube-api-access-5bdfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:30.665373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.665340    7037 generic.go:356] "Generic (PLEG): container finished" podID="a1807b61-e196-46de-b287-0fc7ce0f42bb" containerID="f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6" exitCode=0
Oct 30 21:23:30.665630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.665607    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" event={"ID":"a1807b61-e196-46de-b287-0fc7ce0f42bb","Type":"ContainerDied","Data":"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"}
Oct 30 21:23:30.665744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.665679    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7" event={"ID":"a1807b61-e196-46de-b287-0fc7ce0f42bb","Type":"ContainerDied","Data":"7633c2068febfad53a5ef0631f4ab1b3b745c54b6ec298c6e81faabdb5d5e1fc"}
Oct 30 21:23:30.665744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.665707    7037 scope.go:122] "RemoveContainer" containerID="f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"
Oct 30 21:23:30.665866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.665845    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"
Oct 30 21:23:30.687540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.687517    7037 scope.go:122] "RemoveContainer" containerID="f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"
Oct 30 21:23:30.687883 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:30.687856    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6\": container with ID starting with f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6 not found: ID does not exist" containerID="f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"
Oct 30 21:23:30.687986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.687892    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6"} err="failed to get container status \"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6\": rpc error: code = NotFound desc = could not find container \"f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6\": container with ID starting with f48df409f9aa40254fd82d56da82eb970d5f5ba90d8f51a0e8b6cb49aee60fa6 not found: ID does not exist"
Oct 30 21:23:30.729226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.729141    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:23:30.734282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.734254    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-whcq7"]
Oct 30 21:23:30.750774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.750736    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:30.750774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.750767    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1807b61-e196-46de-b287-0fc7ce0f42bb-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:30.750954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:30.750782    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdfh\" (UniqueName: \"kubernetes.io/projected/a1807b61-e196-46de-b287-0fc7ce0f42bb-kube-api-access-5bdfh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:32.708732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:32.708700    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1807b61-e196-46de-b287-0fc7ce0f42bb" path="/var/lib/kubelet/pods/a1807b61-e196-46de-b287-0fc7ce0f42bb/volumes"
Oct 30 21:23:33.680967 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:33.680927    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:23:34.847080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:34.847044    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:23:34.898161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:34.898124    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:23:36.332120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.332033    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:23:36.333358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333328    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333358    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333390    7037 cpu_manager.go:414] "containerMap: removing container" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333398    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333419    7037 cpu_manager.go:414] "containerMap: removing container" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333429    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333460    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy"
Oct 30 21:23:36.333484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333467    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy"
Oct 30 21:23:36.333808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333528    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a1807b61-e196-46de-b287-0fc7ce0f42bb" containerName="maas-api"
Oct 30 21:23:36.333808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333536    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1807b61-e196-46de-b287-0fc7ce0f42bb" containerName="maas-api"
Oct 30 21:23:36.333808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333757    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9156b2d8-9d48-4320-817e-29fa49050c5d" containerName="istio-proxy"
Oct 30 21:23:36.333808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333781    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1807b61-e196-46de-b287-0fc7ce0f42bb" containerName="maas-api"
Oct 30 21:23:36.333808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333798    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ac3d5-ebdf-45b5-aab5-5f14be8e2aa2" containerName="istio-proxy"
Oct 30 21:23:36.333973 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333809    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="be596f6c-0bd4-4ae7-9075-1040c449f54d" containerName="istio-proxy"
Oct 30 21:23:36.333973 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.333821    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fd1d12-f119-4cce-9e58-53222ca65c74" containerName="istio-proxy"
Oct 30 21:23:36.340310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.340289    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.374511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.360545    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:23:36.374511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.364259    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-95692f-dockercfg-wd7wz\""
Oct 30 21:23:36.374511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.364330    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-95692f\""
Oct 30 21:23:36.434197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.434166    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:23:36.439120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.439097    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.469880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.465257    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:23:36.469880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.466141    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.469880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.466529    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhvrx\" (UniqueName: \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.580542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.572505    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.580542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.572619    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hsd\" (UniqueName: \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.580542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.572728    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhvrx\" (UniqueName: \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.580542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.572886    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.580542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.574019    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.647425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.647338    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhvrx\" (UniqueName: \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\") pod \"payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.666286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.665994    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:36.718085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.717656    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.718085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.717774    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hsd\" (UniqueName: \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.725411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.725376    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.795115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.792221    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hsd\" (UniqueName: \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\") pod \"payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:36.795115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:36.792594    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:37.064275 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:37.064233    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265bdf6b_1fee_496e_ba32_a516560d5fb1.slice/crio-bbf8c940a5813ded507804a19407c827308b4758c1aab0a1094e67c506f18156 WatchSource:0}: Error finding container bbf8c940a5813ded507804a19407c827308b4758c1aab0a1094e67c506f18156: Status 404 returned error can't find the container with id bbf8c940a5813ded507804a19407c827308b4758c1aab0a1094e67c506f18156
Oct 30 21:23:37.065527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.065095    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:23:37.158900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.157710    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:23:37.162189 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:37.162083    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08137a39_98c2_4895_afb4_6cfcccc27932.slice/crio-2f08c8c12411639f1b60c3a268593dde8265592e484531fac67c80eb6615ed58 WatchSource:0}: Error finding container 2f08c8c12411639f1b60c3a268593dde8265592e484531fac67c80eb6615ed58: Status 404 returned error can't find the container with id 2f08c8c12411639f1b60c3a268593dde8265592e484531fac67c80eb6615ed58
Oct 30 21:23:37.768966 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.768926    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" event={"ID":"265bdf6b-1fee-496e-ba32-a516560d5fb1","Type":"ContainerStarted","Data":"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"}
Oct 30 21:23:37.769993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.769062    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" event={"ID":"265bdf6b-1fee-496e-ba32-a516560d5fb1","Type":"ContainerStarted","Data":"bbf8c940a5813ded507804a19407c827308b4758c1aab0a1094e67c506f18156"}
Oct 30 21:23:37.769993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.769183    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:37.772128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.772086    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" event={"ID":"08137a39-98c2-4895-afb4-6cfcccc27932","Type":"ContainerStarted","Data":"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"}
Oct 30 21:23:37.772232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.772151    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" event={"ID":"08137a39-98c2-4895-afb4-6cfcccc27932","Type":"ContainerStarted","Data":"2f08c8c12411639f1b60c3a268593dde8265592e484531fac67c80eb6615ed58"}
Oct 30 21:23:37.772232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.772178    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:37.810590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.810540    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" podStartSLOduration=1.8105268799999998 podStartE2EDuration="1.81052688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:37.808873777 +0000 UTC m=+1361.220413213" watchObservedRunningTime="2026-10-30 21:23:37.81052688 +0000 UTC m=+1361.222066310"
Oct 30 21:23:37.854477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:37.854428    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" podStartSLOduration=1.854414638 podStartE2EDuration="1.854414638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:37.8483429 +0000 UTC m=+1361.259882336" watchObservedRunningTime="2026-10-30 21:23:37.854414638 +0000 UTC m=+1361.265954074"
Oct 30 21:23:39.847530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:39.847482    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" podUID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" containerName="payload-processing" containerID="cri-o://4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb" gracePeriod=25
Oct 30 21:23:39.898623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:39.898587    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" podUID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" containerName="payload-pre-processing" containerID="cri-o://4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d" gracePeriod=25
Oct 30 21:23:40.235656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.235631    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:23:40.239153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.239132    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:23:40.286863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.286814    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\") pod \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") "
Oct 30 21:23:40.287051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.286962    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\" (UniqueName: \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\") pod \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") "
Oct 30 21:23:40.287051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.287048    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\" (UniqueName: \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\") pod \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\" (UID: \"5fdbb835-aabf-4b41-a4ea-297b60a0f30d\") "
Oct 30 21:23:40.287360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.287338    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\") pod \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\" (UID: \"fa44567f-a2ad-4f81-aeb1-1c3669bedb69\") "
Oct 30 21:23:40.289472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.289437    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume" pod "fa44567f-a2ad-4f81-aeb1-1c3669bedb69" (UID: "fa44567f-a2ad-4f81-aeb1-1c3669bedb69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:40.290039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.289934    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume" pod "5fdbb835-aabf-4b41-a4ea-297b60a0f30d" (UID: "5fdbb835-aabf-4b41-a4ea-297b60a0f30d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:40.293220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.293197    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl" pod "5fdbb835-aabf-4b41-a4ea-297b60a0f30d" (UID: "5fdbb835-aabf-4b41-a4ea-297b60a0f30d"). InnerVolumeSpecName "kube-api-access-nmvtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:40.293392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.293371    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s" pod "fa44567f-a2ad-4f81-aeb1-1c3669bedb69" (UID: "fa44567f-a2ad-4f81-aeb1-1c3669bedb69"). InnerVolumeSpecName "kube-api-access-glz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:40.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.390539    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz5s\" (UniqueName: \"kubernetes.io/projected/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-kube-api-access-glz5s\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:40.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.390572    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvtl\" (UniqueName: \"kubernetes.io/projected/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-kube-api-access-nmvtl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:40.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.390581    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44567f-a2ad-4f81-aeb1-1c3669bedb69-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:40.390635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.390591    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fdbb835-aabf-4b41-a4ea-297b60a0f30d-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:40.420021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.419990    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:23:40.420825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420810    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" containerName="payload-pre-processing"
Oct 30 21:23:40.420880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420826    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" containerName="payload-pre-processing"
Oct 30 21:23:40.420880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420866    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" containerName="payload-processing"
Oct 30 21:23:40.420880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420871    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" containerName="payload-processing"
Oct 30 21:23:40.420985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420976    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" containerName="payload-processing"
Oct 30 21:23:40.421019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.420995    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" containerName="payload-pre-processing"
Oct 30 21:23:40.425076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.425055    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.443256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.443226    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c1da2183-dockercfg-njrlb\""
Oct 30 21:23:40.443256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.443238    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c1da2183\""
Oct 30 21:23:40.443998 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.443968    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:23:40.526558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.524243    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:23:40.540511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.537459    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.551255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.550373    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:23:40.629602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.607703    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.629602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.607789    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjxb\" (UniqueName: \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.629602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.607866    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tmf\" (UniqueName: \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.629602 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.607986    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.711721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.710520    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.711721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.710624    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjxb\" (UniqueName: \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.711721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.710707    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tmf\" (UniqueName: \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.711721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.710836    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.713222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.712513    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.717964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.713537    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.736257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.736232    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjxb\" (UniqueName: \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\") pod \"payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.736515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.736473    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tmf\" (UniqueName: \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\") pod \"payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") " pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.740298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.740275    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:40.807531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.805451    7037 generic.go:356] "Generic (PLEG): container finished" podID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" containerID="4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb" exitCode=0
Oct 30 21:23:40.807531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.805678    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" event={"ID":"5fdbb835-aabf-4b41-a4ea-297b60a0f30d","Type":"ContainerDied","Data":"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"}
Oct 30 21:23:40.807531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.805735    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm" event={"ID":"5fdbb835-aabf-4b41-a4ea-297b60a0f30d","Type":"ContainerDied","Data":"b54a8cec02d5e2fdc7056381f519d44af706c94a8a35dddf364382b01d4ff6b2"}
Oct 30 21:23:40.807531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.805764    7037 scope.go:122] "RemoveContainer" containerID="4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"
Oct 30 21:23:40.807531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.805942    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"
Oct 30 21:23:40.811710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.811546    7037 generic.go:356] "Generic (PLEG): container finished" podID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" containerID="4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d" exitCode=0
Oct 30 21:23:40.811710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.811592    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" event={"ID":"fa44567f-a2ad-4f81-aeb1-1c3669bedb69","Type":"ContainerDied","Data":"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"}
Oct 30 21:23:40.811710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.811597    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"
Oct 30 21:23:40.811710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.811630    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g" event={"ID":"fa44567f-a2ad-4f81-aeb1-1c3669bedb69","Type":"ContainerDied","Data":"3e743fb9621a04ef4704eb94f55144a7e4eee3b9167ecca009f0ef2415e2fb7d"}
Oct 30 21:23:40.834002 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.833826    7037 scope.go:122] "RemoveContainer" containerID="4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"
Oct 30 21:23:40.834343 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:40.834316    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb\": container with ID starting with 4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb not found: ID does not exist" containerID="4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"
Oct 30 21:23:40.834453 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.834353    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb"} err="failed to get container status \"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb\": rpc error: code = NotFound desc = could not find container \"4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb\": container with ID starting with 4bf554e42ecde56262689a5654713a8890c8801d5394c3081a4198364e09cffb not found: ID does not exist"
Oct 30 21:23:40.834453 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.834375    7037 scope.go:122] "RemoveContainer" containerID="4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"
Oct 30 21:23:40.868388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.868145    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:40.899553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.899380    7037 scope.go:122] "RemoveContainer" containerID="4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"
Oct 30 21:23:40.899959 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:40.899873    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d\": container with ID starting with 4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d not found: ID does not exist" containerID="4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"
Oct 30 21:23:40.899959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.899915    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d"} err="failed to get container status \"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d\": rpc error: code = NotFound desc = could not find container \"4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d\": container with ID starting with 4def44784bb11d8877efb7b92a99eb62a9fc58616a23623a4b1141114d39a83d not found: ID does not exist"
Oct 30 21:23:40.909562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.905372    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:23:40.910559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.910531    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-59fb17-649c79cc74-jrnbm"]
Oct 30 21:23:40.948423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.948381    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:23:40.955277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:40.955224    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-59fb17-89bc69f67-56k2g"]
Oct 30 21:23:41.065362 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:41.065332    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75579726_a0de_4f74_9ac1_f88f8ef6ad32.slice/crio-8995945fab76d962def7b93f48f56d83b3db8b0461e88c28c21908be4c4bf478 WatchSource:0}: Error finding container 8995945fab76d962def7b93f48f56d83b3db8b0461e88c28c21908be4c4bf478: Status 404 returned error can't find the container with id 8995945fab76d962def7b93f48f56d83b3db8b0461e88c28c21908be4c4bf478
Oct 30 21:23:41.065832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.065807    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:23:41.210864 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:41.210817    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd089c0_79b9_4b34_b6b3_f38d52751496.slice/crio-9f42c62b388d4feaab36fbb0d2659d7c6fbecff670f37e58e07b839720401c0f WatchSource:0}: Error finding container 9f42c62b388d4feaab36fbb0d2659d7c6fbecff670f37e58e07b839720401c0f: Status 404 returned error can't find the container with id 9f42c62b388d4feaab36fbb0d2659d7c6fbecff670f37e58e07b839720401c0f
Oct 30 21:23:41.211095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.211076    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:23:41.391070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.391036    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:23:41.434812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.434777    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:23:41.829790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.829524    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" event={"ID":"75579726-a0de-4f74-9ac1-f88f8ef6ad32","Type":"ContainerStarted","Data":"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"}
Oct 30 21:23:41.829790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.829594    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" event={"ID":"75579726-a0de-4f74-9ac1-f88f8ef6ad32","Type":"ContainerStarted","Data":"8995945fab76d962def7b93f48f56d83b3db8b0461e88c28c21908be4c4bf478"}
Oct 30 21:23:41.829790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.829714    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:41.832145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.832117    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" event={"ID":"abd089c0-79b9-4b34-b6b3-f38d52751496","Type":"ContainerStarted","Data":"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"}
Oct 30 21:23:41.832272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.832163    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" event={"ID":"abd089c0-79b9-4b34-b6b3-f38d52751496","Type":"ContainerStarted","Data":"9f42c62b388d4feaab36fbb0d2659d7c6fbecff670f37e58e07b839720401c0f"}
Oct 30 21:23:41.832272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.832212    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:41.884565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.884511    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" podStartSLOduration=1.8844844790000002 podStartE2EDuration="1.884484479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:41.882907478 +0000 UTC m=+1365.294446913" watchObservedRunningTime="2026-10-30 21:23:41.884484479 +0000 UTC m=+1365.296023914"
Oct 30 21:23:41.928253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:41.928204    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" podStartSLOduration=1.928188933 podStartE2EDuration="1.928188933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:41.92793831 +0000 UTC m=+1365.339477746" watchObservedRunningTime="2026-10-30 21:23:41.928188933 +0000 UTC m=+1365.339728368"
Oct 30 21:23:42.711116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:42.711074    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fdbb835-aabf-4b41-a4ea-297b60a0f30d" path="/var/lib/kubelet/pods/5fdbb835-aabf-4b41-a4ea-297b60a0f30d/volumes"
Oct 30 21:23:42.712149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:42.712123    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa44567f-a2ad-4f81-aeb1-1c3669bedb69" path="/var/lib/kubelet/pods/fa44567f-a2ad-4f81-aeb1-1c3669bedb69/volumes"
Oct 30 21:23:46.125205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.125171    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"]
Oct 30 21:23:46.137465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.137438    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d021d0-5mxxq"]
Oct 30 21:23:46.391570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.391438    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" podUID="7b297685-d673-4246-82c2-61a0c073d078" containerName="payload-processing" containerID="cri-o://147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd" gracePeriod=25
Oct 30 21:23:46.436387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.435601    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" podUID="48bd36e3-c428-4568-aa2c-4711a3990f85" containerName="payload-pre-processing" containerID="cri-o://8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7" gracePeriod=25
Oct 30 21:23:46.712779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.712747    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef8662b-d830-4e0f-af16-472c3439944e" path="/var/lib/kubelet/pods/3ef8662b-d830-4e0f-af16-472c3439944e/volumes"
Oct 30 21:23:46.742269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.742043    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:23:46.791247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.791218    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:23:46.814193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.812995    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\" (UniqueName: \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\") pod \"7b297685-d673-4246-82c2-61a0c073d078\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") "
Oct 30 21:23:46.814193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.813114    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\") pod \"7b297685-d673-4246-82c2-61a0c073d078\" (UID: \"7b297685-d673-4246-82c2-61a0c073d078\") "
Oct 30 21:23:46.817250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.817221    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume" pod "7b297685-d673-4246-82c2-61a0c073d078" (UID: "7b297685-d673-4246-82c2-61a0c073d078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:46.818743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.818717    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np" pod "7b297685-d673-4246-82c2-61a0c073d078" (UID: "7b297685-d673-4246-82c2-61a0c073d078"). InnerVolumeSpecName "kube-api-access-c64np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:46.889423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.889362    7037 generic.go:356] "Generic (PLEG): container finished" podID="7b297685-d673-4246-82c2-61a0c073d078" containerID="147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd" exitCode=0
Oct 30 21:23:46.889684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.889639    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" event={"ID":"7b297685-d673-4246-82c2-61a0c073d078","Type":"ContainerDied","Data":"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"}
Oct 30 21:23:46.889932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.889906    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"
Oct 30 21:23:46.890147 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.890036    7037 scope.go:122] "RemoveContainer" containerID="147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"
Oct 30 21:23:46.890147 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.890043    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q" event={"ID":"7b297685-d673-4246-82c2-61a0c073d078","Type":"ContainerDied","Data":"ee6ac210f273b10150340a8206dd2b1a63e7b127e1a941fe3737c9639ec5abcd"}
Oct 30 21:23:46.892650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.892623    7037 generic.go:356] "Generic (PLEG): container finished" podID="48bd36e3-c428-4568-aa2c-4711a3990f85" containerID="8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7" exitCode=0
Oct 30 21:23:46.892786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.892768    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" event={"ID":"48bd36e3-c428-4568-aa2c-4711a3990f85","Type":"ContainerDied","Data":"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"}
Oct 30 21:23:46.892861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.892816    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz" event={"ID":"48bd36e3-c428-4568-aa2c-4711a3990f85","Type":"ContainerDied","Data":"9e7906ab452cc1d6c2a55d39644fe902d1034a131374c0575eafd4665843c887"}
Oct 30 21:23:46.892951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.892932    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"
Oct 30 21:23:46.907586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.907565    7037 scope.go:122] "RemoveContainer" containerID="147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"
Oct 30 21:23:46.907917 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:46.907890    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd\": container with ID starting with 147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd not found: ID does not exist" containerID="147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"
Oct 30 21:23:46.907975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.907927    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd"} err="failed to get container status \"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd\": rpc error: code = NotFound desc = could not find container \"147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd\": container with ID starting with 147dbcc6669155335c4f9721b33b91bc1d14aba14376506536137885d9f5adfd not found: ID does not exist"
Oct 30 21:23:46.907975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.907951    7037 scope.go:122] "RemoveContainer" containerID="8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"
Oct 30 21:23:46.915622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.915561    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\" (UniqueName: \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\") pod \"48bd36e3-c428-4568-aa2c-4711a3990f85\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") "
Oct 30 21:23:46.915784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.915714    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\") pod \"48bd36e3-c428-4568-aa2c-4711a3990f85\" (UID: \"48bd36e3-c428-4568-aa2c-4711a3990f85\") "
Oct 30 21:23:46.916430 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.916085    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume" pod "48bd36e3-c428-4568-aa2c-4711a3990f85" (UID: "48bd36e3-c428-4568-aa2c-4711a3990f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:46.916671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.916649    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd36e3-c428-4568-aa2c-4711a3990f85-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:46.916755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.916675    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64np\" (UniqueName: \"kubernetes.io/projected/7b297685-d673-4246-82c2-61a0c073d078-kube-api-access-c64np\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:46.916755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.916689    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b297685-d673-4246-82c2-61a0c073d078-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:46.916899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.916879    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr" pod "48bd36e3-c428-4568-aa2c-4711a3990f85" (UID: "48bd36e3-c428-4568-aa2c-4711a3990f85"). InnerVolumeSpecName "kube-api-access-xztbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:46.919314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.919293    7037 scope.go:122] "RemoveContainer" containerID="8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"
Oct 30 21:23:46.919648 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:23:46.919625    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7\": container with ID starting with 8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7 not found: ID does not exist" containerID="8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"
Oct 30 21:23:46.919726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.919653    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7"} err="failed to get container status \"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7\": rpc error: code = NotFound desc = could not find container \"8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7\": container with ID starting with 8c10394a3a38c8e374003841f05a0e862d742dbc884a9e1267f3b2e860f6e4b7 not found: ID does not exist"
Oct 30 21:23:46.962958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.962919    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:23:46.966686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:46.966661    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0e7510-574b9c5d94-hvp2q"]
Oct 30 21:23:47.017747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:47.017709    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztbr\" (UniqueName: \"kubernetes.io/projected/48bd36e3-c428-4568-aa2c-4711a3990f85-kube-api-access-xztbr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:23:47.251040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:47.250999    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:23:47.273401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:47.273368    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0e7510-646866ffb7-t5vrz"]
Oct 30 21:23:48.712410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.712380    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bd36e3-c428-4568-aa2c-4711a3990f85" path="/var/lib/kubelet/pods/48bd36e3-c428-4568-aa2c-4711a3990f85/volumes"
Oct 30 21:23:48.712937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.712923    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b297685-d673-4246-82c2-61a0c073d078" path="/var/lib/kubelet/pods/7b297685-d673-4246-82c2-61a0c073d078/volumes"
Oct 30 21:23:48.785239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.785204    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:23:48.785425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.785271    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:23:48.795352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.795320    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:23:48.796221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796199    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7b297685-d673-4246-82c2-61a0c073d078" containerName="payload-processing"
Oct 30 21:23:48.796221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796220    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b297685-d673-4246-82c2-61a0c073d078" containerName="payload-processing"
Oct 30 21:23:48.796400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796242    7037 cpu_manager.go:414] "containerMap: removing container" podUID="48bd36e3-c428-4568-aa2c-4711a3990f85" containerName="payload-pre-processing"
Oct 30 21:23:48.796400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796251    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bd36e3-c428-4568-aa2c-4711a3990f85" containerName="payload-pre-processing"
Oct 30 21:23:48.796547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796533    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b297685-d673-4246-82c2-61a0c073d078" containerName="payload-processing"
Oct 30 21:23:48.796600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.796554    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bd36e3-c428-4568-aa2c-4711a3990f85" containerName="payload-pre-processing"
Oct 30 21:23:48.802023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.802002    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.830445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.830402    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:23:48.902380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.902314    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ca18b3-openshift-default-dockercfg-kl4ft\""
Oct 30 21:23:48.976696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.976624    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrl2\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.976696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.976675    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.976864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.976795    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.976932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.976920    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.976979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.976965    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.977051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.977030    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.977122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.977107    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.977159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.977141    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:48.977195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:48.977171    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078329    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrl2\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078387    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078428    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078596    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078631    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078704    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078740    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078783    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.078988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.078877    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.079075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.079020    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.079282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.079261    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.079472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.079452    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.079641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.079626    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.081292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.081271    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.081684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.081663    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.103181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.103155    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.103327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.103308    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrl2\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.123232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.123206    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:49.446907 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:49.446872    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47beaf0a_34f3_4571_9b21_32c7083b2d67.slice/crio-8d03eac0f0448c6812e6437070b9b5c8791512b0630064f08d31ffb46b883725 WatchSource:0}: Error finding container 8d03eac0f0448c6812e6437070b9b5c8791512b0630064f08d31ffb46b883725: Status 404 returned error can't find the container with id 8d03eac0f0448c6812e6437070b9b5c8791512b0630064f08d31ffb46b883725
Oct 30 21:23:49.449707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.449670    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:23:49.449809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.449744    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:23:49.449809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.449774    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:23:49.449809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.449785    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:23:49.526716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.526690    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:23:49.531678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.531660    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.554152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.554127    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c31999\""
Oct 30 21:23:49.554349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.554229    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c31999-dockercfg-wnld5\""
Oct 30 21:23:49.563215 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.563186    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:23:49.578634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.578600    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:23:49.595693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.595664    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:23:49.595862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.595794    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.702570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.699280    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.702570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.699340    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm84h\" (UniqueName: \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.702570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.699388    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.702570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.699420    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnwh\" (UniqueName: \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.801909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.801863    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.802420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.801960    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm84h\" (UniqueName: \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.802420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.802074    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.802420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.802120    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnwh\" (UniqueName: \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.803027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.802995    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.803872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.803844    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.823880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.823851    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm84h\" (UniqueName: \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\") pod \"payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.825455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.825431    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnwh\" (UniqueName: \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\") pod \"payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.846778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.846714    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:49.925838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.925756    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:49.979288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.972210    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" event={"ID":"47beaf0a-34f3-4571-9b21-32c7083b2d67","Type":"ContainerStarted","Data":"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"}
Oct 30 21:23:49.979288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:49.972271    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" event={"ID":"47beaf0a-34f3-4571-9b21-32c7083b2d67","Type":"ContainerStarted","Data":"8d03eac0f0448c6812e6437070b9b5c8791512b0630064f08d31ffb46b883725"}
Oct 30 21:23:50.033608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.026749    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podStartSLOduration=2.026728331 podStartE2EDuration="2.026728331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:50.020730603 +0000 UTC m=+1373.432270040" watchObservedRunningTime="2026-10-30 21:23:50.026728331 +0000 UTC m=+1373.438267771"
Oct 30 21:23:50.126190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.125374    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:50.126190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.125670    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:50.126190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.125718    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:50.172295 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:50.172268    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3339715f_18ba_4d60_9f5b_54412d6794e4.slice/crio-79a5bcc324490cd7ffefd927fccdcc42da2da5040a28370a2088c519a3a51444 WatchSource:0}: Error finding container 79a5bcc324490cd7ffefd927fccdcc42da2da5040a28370a2088c519a3a51444: Status 404 returned error can't find the container with id 79a5bcc324490cd7ffefd927fccdcc42da2da5040a28370a2088c519a3a51444
Oct 30 21:23:50.173052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.172975    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:23:50.285910 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:23:50.285873    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50f2743_a063_469e_9dba_da4a43ac857a.slice/crio-d17550b7856613fce588b20448d2e1199ee7e561354bf2e72d501c608ebb1610 WatchSource:0}: Error finding container d17550b7856613fce588b20448d2e1199ee7e561354bf2e72d501c608ebb1610: Status 404 returned error can't find the container with id d17550b7856613fce588b20448d2e1199ee7e561354bf2e72d501c608ebb1610
Oct 30 21:23:50.285910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.285890    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:23:50.982073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.982033    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" event={"ID":"a50f2743-a063-469e-9dba-da4a43ac857a","Type":"ContainerStarted","Data":"8e7a9055dbfaea49d11e677a633cf94da856107539bc2454deb2cc055a1365de"}
Oct 30 21:23:50.982582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.982095    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" event={"ID":"a50f2743-a063-469e-9dba-da4a43ac857a","Type":"ContainerStarted","Data":"d17550b7856613fce588b20448d2e1199ee7e561354bf2e72d501c608ebb1610"}
Oct 30 21:23:50.982582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.982124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:23:50.983823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.983793    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" event={"ID":"3339715f-18ba-4d60-9f5b-54412d6794e4","Type":"ContainerStarted","Data":"357dffac3d5f6be4b1c73e0062d8cc230b11c68e6376295dbf4f28b55f2ff0a1"}
Oct 30 21:23:50.983945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.983840    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" event={"ID":"3339715f-18ba-4d60-9f5b-54412d6794e4","Type":"ContainerStarted","Data":"79a5bcc324490cd7ffefd927fccdcc42da2da5040a28370a2088c519a3a51444"}
Oct 30 21:23:50.983945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:50.983868    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:23:51.050738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:51.050669    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" podStartSLOduration=2.050645692 podStartE2EDuration="2.050645692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:51.044811924 +0000 UTC m=+1374.456351354" watchObservedRunningTime="2026-10-30 21:23:51.050645692 +0000 UTC m=+1374.462185130"
Oct 30 21:23:51.089306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:51.089255    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" podStartSLOduration=2.089241243 podStartE2EDuration="2.089241243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:51.085968597 +0000 UTC m=+1374.497508036" watchObservedRunningTime="2026-10-30 21:23:51.089241243 +0000 UTC m=+1374.500780657"
Oct 30 21:23:51.148692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:51.148647    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:51.148894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:51.148720    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:52.131512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:52.129257    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:52.131512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:52.129322    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:52.851121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:52.851083    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:23:52.854051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:52.854026    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:23:53.135566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:53.135199    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:53.135566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:53.135257    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:54.142514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:54.131832    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:54.142514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:54.131892    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:55.144906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:55.144870    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 30 21:23:55.145546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:55.145513    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 30 21:23:56.275512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:56.274849    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:56.277809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:56.276173    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:56.279045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:56.279024    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:23:57.738734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:57.738701    7037 scope.go:122] "RemoveContainer" containerID="4a4877a09e5e3c48d187a242b7ccc01f6d5a02ef87800778fbb38727e538b047"
Oct 30 21:23:57.750099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:57.750076    7037 scope.go:122] "RemoveContainer" containerID="dd427115b23a20919297e956798f5281bb38870b98f11022b03b14b4138a3f5c"
Oct 30 21:23:57.759770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:23:57.759748    7037 scope.go:122] "RemoveContainer" containerID="bc724c043029332435b9d6ffad7eae20eec6acfc9a08928cacfd965c40ee8db6"
Oct 30 21:24:01.996618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:01.996581    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:24:01.997118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:01.996735    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:24:03.673404 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:03.673360    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:24:09.009720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.009686    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:24:09.016654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.016631    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.035422 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.035389    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:24:09.041186 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.041154    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-517366\""
Oct 30 21:24:09.041916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.041895    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-517366-dockercfg-h5jgn\""
Oct 30 21:24:09.061399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.059272    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:24:09.067365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.067337    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.080964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.080931    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdbw\" (UniqueName: \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.081103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.081022    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:24:09.081103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.081062    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.182787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.182751    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdbw\" (UniqueName: \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.182980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.182900    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.183079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.183063    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.183146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.183101    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnpk\" (UniqueName: \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.183627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.183607    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.203472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.203438    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdbw\" (UniqueName: \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\") pod \"payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") " pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.284416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.284338    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.284416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.284386    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnpk\" (UniqueName: \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.285134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.285112    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.306583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.306553    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnpk\" (UniqueName: \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\") pod \"payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.332227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.332194    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:09.387038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.387000    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:09.656293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.656054    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:24:09.661170 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:09.661140    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5706d3_f5ac_44bd_902d_e78fd5ba3e11.slice/crio-971313627c0d266f235259d6fb206e7d8bfefa68c64e6b190d7743f3b7c886a0 WatchSource:0}: Error finding container 971313627c0d266f235259d6fb206e7d8bfefa68c64e6b190d7743f3b7c886a0: Status 404 returned error can't find the container with id 971313627c0d266f235259d6fb206e7d8bfefa68c64e6b190d7743f3b7c886a0
Oct 30 21:24:09.865743 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:09.865710    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942b1948_a3d8_4538_903b_c7ce9179dcfb.slice/crio-f0ccc9e31223c274c11f97e6e1f503f9bfdf0b545eb8f7cba02cf1ad4c3c9c4d WatchSource:0}: Error finding container f0ccc9e31223c274c11f97e6e1f503f9bfdf0b545eb8f7cba02cf1ad4c3c9c4d: Status 404 returned error can't find the container with id f0ccc9e31223c274c11f97e6e1f503f9bfdf0b545eb8f7cba02cf1ad4c3c9c4d
Oct 30 21:24:09.867317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:09.867288    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:24:10.171006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.170910    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" event={"ID":"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11","Type":"ContainerStarted","Data":"00369b47fa3c84b0ab76d0951e839fd216c360b08889a8a0358ca02ea74838c1"}
Oct 30 21:24:10.171006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.170966    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" event={"ID":"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11","Type":"ContainerStarted","Data":"971313627c0d266f235259d6fb206e7d8bfefa68c64e6b190d7743f3b7c886a0"}
Oct 30 21:24:10.171006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.170994    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:10.172503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.172466    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" event={"ID":"942b1948-a3d8-4538-903b-c7ce9179dcfb","Type":"ContainerStarted","Data":"6c0d3c01da2ce04d57b1955027e6368b01b1532239e111fd3426f64397b5afe4"}
Oct 30 21:24:10.172614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.172533    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" event={"ID":"942b1948-a3d8-4538-903b-c7ce9179dcfb","Type":"ContainerStarted","Data":"f0ccc9e31223c274c11f97e6e1f503f9bfdf0b545eb8f7cba02cf1ad4c3c9c4d"}
Oct 30 21:24:10.172614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.172558    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:10.237810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.237761    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" podStartSLOduration=2.237748235 podStartE2EDuration="2.237748235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.235528702 +0000 UTC m=+1393.647068132" watchObservedRunningTime="2026-10-30 21:24:10.237748235 +0000 UTC m=+1393.649287672"
Oct 30 21:24:10.288242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:10.288198    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" podStartSLOduration=1.288186331 podStartE2EDuration="1.288186331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.287521068 +0000 UTC m=+1393.699060505" watchObservedRunningTime="2026-10-30 21:24:10.288186331 +0000 UTC m=+1393.699725766"
Oct 30 21:24:13.593987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.593952    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:24:13.600005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.599979    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.618154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.618127    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:24:13.624969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.624942    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1da2183-serving-cert\""
Oct 30 21:24:13.654352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.654321    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.654573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.654478    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgzg\" (UniqueName: \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.654675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.654644    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.654786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.654770    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.755875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.755832    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.756071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.756002    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.756293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.756226    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.756442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.756419    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgzg\" (UniqueName: \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.757552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.757531    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.758461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.758438    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.759346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.759327    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.781813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.781789    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgzg\" (UniqueName: \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\") pod \"maas-api-e2e-mt-c1da2183-66768659c7-vfrh8\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:13.918906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:13.918803    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:14.264740 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:14.264705    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode671c7b3_4a88_43f1_8729_88e447654cc7.slice/crio-e11c0a000fc9437fa6b9b3b915360fc59a4c665117a299cc0a64b09ae1de98b7 WatchSource:0}: Error finding container e11c0a000fc9437fa6b9b3b915360fc59a4c665117a299cc0a64b09ae1de98b7: Status 404 returned error can't find the container with id e11c0a000fc9437fa6b9b3b915360fc59a4c665117a299cc0a64b09ae1de98b7
Oct 30 21:24:14.266404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:14.266375    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:24:15.215866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:15.215828    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" event={"ID":"e671c7b3-4a88-43f1-8729-88e447654cc7","Type":"ContainerStarted","Data":"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"}
Oct 30 21:24:15.216259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:15.215883    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" event={"ID":"e671c7b3-4a88-43f1-8729-88e447654cc7","Type":"ContainerStarted","Data":"e11c0a000fc9437fa6b9b3b915360fc59a4c665117a299cc0a64b09ae1de98b7"}
Oct 30 21:24:15.216259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:15.215913    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:16.778979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.778922    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" podStartSLOduration=3.471021085 podStartE2EDuration="3.778902141s" totalImagesPullingTime="307.881056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:15.261267701 +0000 UTC m=+1398.672807149" watchObservedRunningTime="2026-10-30 21:24:16.778902141 +0000 UTC m=+1400.190441578"
Oct 30 21:24:16.781520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.781478    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:24:16.786469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.786449    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:16.807765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.807736    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:24:16.814692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.814666    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-95692f-serving-cert\""
Oct 30 21:24:16.913052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.913019    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxwm\" (UniqueName: \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:16.913379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.913353    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:16.913483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.913401    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:16.913620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:16.913535    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.015073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.015040    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.015261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.015083    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.015261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.015114    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.015261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.015192    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxwm\" (UniqueName: \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.015824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.015795    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.017564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.017541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.017669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.017582    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.036037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.035921    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxwm\" (UniqueName: \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\") pod \"maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.102878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.102832    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:17.431612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:17.431580    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:24:17.437704 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:17.437674    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9b6918_d7ce_4a16_b058_298e2f660177.slice/crio-5fd5086f902b0beb1c183fb589bcde06c1370f319049e1d71487f7dc66dc8605 WatchSource:0}: Error finding container 5fd5086f902b0beb1c183fb589bcde06c1370f319049e1d71487f7dc66dc8605: Status 404 returned error can't find the container with id 5fd5086f902b0beb1c183fb589bcde06c1370f319049e1d71487f7dc66dc8605
Oct 30 21:24:18.244303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.244262    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" event={"ID":"ce9b6918-d7ce-4a16-b058-298e2f660177","Type":"ContainerStarted","Data":"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"}
Oct 30 21:24:18.244739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.244331    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" event={"ID":"ce9b6918-d7ce-4a16-b058-298e2f660177","Type":"ContainerStarted","Data":"5fd5086f902b0beb1c183fb589bcde06c1370f319049e1d71487f7dc66dc8605"}
Oct 30 21:24:18.244739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.244408    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:18.289076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.289026    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" podStartSLOduration=1.9401041829999999 podStartE2EDuration="2.289014275s" totalImagesPullingTime="348.910092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:18.285965217 +0000 UTC m=+1401.697504655" watchObservedRunningTime="2026-10-30 21:24:18.289014275 +0000 UTC m=+1401.700553705"
Oct 30 21:24:18.940512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.923216    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:24:18.940512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.929482    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:18.971674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:18.967832    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:24:19.059850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.059816    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.059862    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.059936    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.059962    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.060132    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsz4\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.060175    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.060256    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.060278    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.060343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.060332    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161315    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161370    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161405    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161434    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161602    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsz4\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161671    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161726    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161752    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.161792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161789    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.162054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.161812    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.162054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.162029    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.162229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.162194    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.162292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.162230    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.162343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.162281    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.163734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.163714    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.164144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.164127    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.181142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.181121    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsz4\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.181249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.181227    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.279817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.279736    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:19.619223 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:19.619192    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f733c03_8ae2_485c_90a3_e8d96a82753c.slice/crio-eca18e63d3d946cbf22df3451f510067f77e1ea549aa23bd229a633db5b84214 WatchSource:0}: Error finding container eca18e63d3d946cbf22df3451f510067f77e1ea549aa23bd229a633db5b84214: Status 404 returned error can't find the container with id eca18e63d3d946cbf22df3451f510067f77e1ea549aa23bd229a633db5b84214
Oct 30 21:24:19.622880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.622817    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:24:19.623148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.623119    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:24:19.623368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.623341    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:24:19.626133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:19.626103    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:24:20.274915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.274862    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" event={"ID":"1f733c03-8ae2-485c-90a3-e8d96a82753c","Type":"ContainerStarted","Data":"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"}
Oct 30 21:24:20.275133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.274947    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" event={"ID":"1f733c03-8ae2-485c-90a3-e8d96a82753c","Type":"ContainerStarted","Data":"eca18e63d3d946cbf22df3451f510067f77e1ea549aa23bd229a633db5b84214"}
Oct 30 21:24:20.280436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.280409    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:20.282382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.282359    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:20.282505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.282403    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:20.352735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:20.352663    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podStartSLOduration=2.3526305069999998 podStartE2EDuration="2.352630507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:20.340944474 +0000 UTC m=+1403.752483906" watchObservedRunningTime="2026-10-30 21:24:20.352630507 +0000 UTC m=+1403.764169944"
Oct 30 21:24:21.183257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:21.183227    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:24:21.183460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:21.183299    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:24:21.256191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:21.256163    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:24:21.282266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:21.280856    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:21.282266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:21.280917    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:22.281011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:22.280974    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:22.281207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:22.281033    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:23.280479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:23.280441    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:23.280892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:23.280523    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:24.258376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:24.258343    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:24:24.280775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:24.280735    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:24.281220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:24.280792    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:25.280107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:25.280069    7037 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:24:25.280289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:25.280135    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 21:24:26.285307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.285275    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:26.285813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.285699    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:26.286415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.286392    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:24:26.300315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.300283    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:24:26.306365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.306339    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.323473 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.323446    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c31999-serving-cert\""
Oct 30 21:24:26.326774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.326746    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:24:26.420370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.420335    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.420645 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.420622    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.420786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.420672    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5dpz\" (UniqueName: \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.420850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.420783    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.522901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.522871    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.523092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.522925    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.523092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.522952    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5dpz\" (UniqueName: \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.523092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.523019    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.523793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.523764    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.525699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.525674    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.525810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.525764    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.542384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.542325    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5dpz\" (UniqueName: \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\") pod \"maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.624561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.624209    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:26.942638 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:26.942609    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc706a8c9_0ed3_4c71_9d21_556293c78063.slice/crio-2fb9192f4d7368789e3c4fdfe8ab8b1a7560ad589ef0d85f8afd8e9ed3d63965 WatchSource:0}: Error finding container 2fb9192f4d7368789e3c4fdfe8ab8b1a7560ad589ef0d85f8afd8e9ed3d63965: Status 404 returned error can't find the container with id 2fb9192f4d7368789e3c4fdfe8ab8b1a7560ad589ef0d85f8afd8e9ed3d63965
Oct 30 21:24:26.944294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:26.944269    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:24:27.385544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:27.385018    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" event={"ID":"c706a8c9-0ed3-4c71-9d21-556293c78063","Type":"ContainerStarted","Data":"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"}
Oct 30 21:24:27.385544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:27.385089    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" event={"ID":"c706a8c9-0ed3-4c71-9d21-556293c78063","Type":"ContainerStarted","Data":"2fb9192f4d7368789e3c4fdfe8ab8b1a7560ad589ef0d85f8afd8e9ed3d63965"}
Oct 30 21:24:27.385544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:27.385124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:27.429513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:27.429449    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" podStartSLOduration=1.133062032 podStartE2EDuration="1.429436537s" totalImagesPullingTime="296.374505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:27.427095726 +0000 UTC m=+1410.838635188" watchObservedRunningTime="2026-10-30 21:24:27.429436537 +0000 UTC m=+1410.840975972"
Oct 30 21:24:30.890357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:30.890320    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:24:30.890846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:30.890608    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" podUID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" containerName="maas-api" containerID="cri-o://82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf" gracePeriod=30
Oct 30 21:24:31.233956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.233929    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:24:31.411226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.411190    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\") pod \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") "
Oct 30 21:24:31.411446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.411322    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\") pod \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") "
Oct 30 21:24:31.411817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.411792    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\") pod \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") "
Oct 30 21:24:31.412174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.412148    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca" pod "3f860b90-ffa6-431c-af8f-b3e1cb6c7599" (UID: "3f860b90-ffa6-431c-af8f-b3e1cb6c7599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:31.412341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.412318    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\" (UniqueName: \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\") pod \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\" (UID: \"3f860b90-ffa6-431c-af8f-b3e1cb6c7599\") "
Oct 30 21:24:31.413246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.413217    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:31.414595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.414566    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs" pod "3f860b90-ffa6-431c-af8f-b3e1cb6c7599" (UID: "3f860b90-ffa6-431c-af8f-b3e1cb6c7599"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:31.418955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.418917    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls" pod "3f860b90-ffa6-431c-af8f-b3e1cb6c7599" (UID: "3f860b90-ffa6-431c-af8f-b3e1cb6c7599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:31.419076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.419030    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf" pod "3f860b90-ffa6-431c-af8f-b3e1cb6c7599" (UID: "3f860b90-ffa6-431c-af8f-b3e1cb6c7599"). InnerVolumeSpecName "kube-api-access-9m7hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:31.424140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.424107    7037 generic.go:356] "Generic (PLEG): container finished" podID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" containerID="82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf" exitCode=0
Oct 30 21:24:31.424253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.424164    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"
Oct 30 21:24:31.424426 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.424406    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" event={"ID":"3f860b90-ffa6-431c-af8f-b3e1cb6c7599","Type":"ContainerDied","Data":"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"}
Oct 30 21:24:31.424541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.424456    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj" event={"ID":"3f860b90-ffa6-431c-af8f-b3e1cb6c7599","Type":"ContainerDied","Data":"126ef6f54aed0daeeef3348e969115af4cecbcbc2d26b910490cd5eccabc3314"}
Oct 30 21:24:31.424541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.424504    7037 scope.go:122] "RemoveContainer" containerID="82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"
Oct 30 21:24:31.466862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.466825    7037 scope.go:122] "RemoveContainer" containerID="82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"
Oct 30 21:24:31.467249 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:31.467181    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf\": container with ID starting with 82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf not found: ID does not exist" containerID="82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"
Oct 30 21:24:31.467249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.467218    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf"} err="failed to get container status \"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf\": rpc error: code = NotFound desc = could not find container \"82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf\": container with ID starting with 82d3789ad1d8d1d860690164a2b29c161b803b3e09185a5ca8980bce6f9bdcbf not found: ID does not exist"
Oct 30 21:24:31.506641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.506611    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:24:31.512016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.511972    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-59fb17-86f6b7d6f5-9n2cj"]
Oct 30 21:24:31.515403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.515379    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:31.515403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.515402    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m7hf\" (UniqueName: \"kubernetes.io/projected/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-kube-api-access-9m7hf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:31.515403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:31.515411    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f860b90-ffa6-431c-af8f-b3e1cb6c7599-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:32.710987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:32.710953    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" path="/var/lib/kubelet/pods/3f860b90-ffa6-431c-af8f-b3e1cb6c7599/volumes"
Oct 30 21:24:33.397592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:33.397554    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:24:33.674040 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:33.673954    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:24:38.699142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:38.699103    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:24:38.699687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:38.699358    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy" containerID="cri-o://cf6cd9dcab388aa47c67dcdc417a9976bbd1f6638afa3d000be01af3ad1a98e6" gracePeriod=30
Oct 30 21:24:44.788516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:44.778567    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 30 21:24:45.026825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.026779    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:24:45.080726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.080678    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:45.105054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.085443    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:24:45.105054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.086851    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" containerName="maas-api"
Oct 30 21:24:45.105054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.086871    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" containerName="maas-api"
Oct 30 21:24:45.105054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.087133    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f860b90-ffa6-431c-af8f-b3e1cb6c7599" containerName="maas-api"
Oct 30 21:24:45.105054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.097161    7037 generic.go:356] "Generic (PLEG): container finished" podID="723ff275-9ef2-426a-979e-04728732bb1e" containerID="cf6cd9dcab388aa47c67dcdc417a9976bbd1f6638afa3d000be01af3ad1a98e6" exitCode=0
Oct 30 21:24:45.124475 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.123957    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:24:45.193524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.189766    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:24:45.292412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.292366    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:24:45.386084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.345519    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.391922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.391893    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.397514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.392217    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.397815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.397792    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.397948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.376907    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:45.398092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.398075    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.407637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.407607    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.438144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.438111    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.438735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.438712    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.438979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.438961    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\") pod \"723ff275-9ef2-426a-979e-04728732bb1e\" (UID: \"723ff275-9ef2-426a-979e-04728732bb1e\") "
Oct 30 21:24:45.464514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.439460    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.471754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.471718    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.554998 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.554954    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clql4\" (UniqueName: \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.555324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.555299    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.571551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.411751    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:45.604517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.198672    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.372832    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" event={"ID":"723ff275-9ef2-426a-979e-04728732bb1e","Type":"ContainerDied","Data":"cf6cd9dcab388aa47c67dcdc417a9976bbd1f6638afa3d000be01af3ad1a98e6"}
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.638941    7037 scope.go:122] "RemoveContainer" containerID="cf6cd9dcab388aa47c67dcdc417a9976bbd1f6638afa3d000be01af3ad1a98e6"
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.391723    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.396732    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.437926    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.438630    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.447368    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.577479    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "kube-api-access-dqnk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:45.653243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.622689    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.654714    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.655048    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-517366-serving-cert\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.657056    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clql4\" (UniqueName: \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.657125    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.657242    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.657375    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:45.667763    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-517366-serving-cert: secret "maas-api-e2e-api-b-517366-serving-cert" not found
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:45.667836    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls podName:4dfb9445-d7c7-4a60-a505-4105787b08dd nodeName:}" failed. No retries permitted until 2026-10-30 21:24:46.167814694 +0000 UTC m=+1429.579354123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls") pod "maas-api-e2e-api-b-517366-84577f449d-cfsjc" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd") : secret "maas-api-e2e-api-b-517366-serving-cert" not found
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.671267    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678723    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678753    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678768    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678787    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723ff275-9ef2-426a-979e-04728732bb1e-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678801    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678814    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678830    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqnk7\" (UniqueName: \"kubernetes.io/projected/723ff275-9ef2-426a-979e-04728732bb1e-kube-api-access-dqnk7\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.678898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.678859    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723ff275-9ef2-426a-979e-04728732bb1e-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:45.718656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.703132    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.740601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.740559    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo" pod "723ff275-9ef2-426a-979e-04728732bb1e" (UID: "723ff275-9ef2-426a-979e-04728732bb1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:45.794023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.779368    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clql4\" (UniqueName: \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:45.812774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:45.811180    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723ff275-9ef2-426a-979e-04728732bb1e-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:46.114795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.114759    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"
Oct 30 21:24:46.114986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.114767    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7" event={"ID":"723ff275-9ef2-426a-979e-04728732bb1e","Type":"ContainerDied","Data":"49d219dc03b61dc02122816bc3f2be15cdb82bee40b1f44ee68e1d1e82ca58c0"}
Oct 30 21:24:46.206030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.205993    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:24:46.213023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.212985    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-gm5q7"]
Oct 30 21:24:46.218297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.217654    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:46.218297 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:46.217959    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-517366-serving-cert: secret "maas-api-e2e-api-b-517366-serving-cert" not found
Oct 30 21:24:46.218297 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:46.218013    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls podName:4dfb9445-d7c7-4a60-a505-4105787b08dd nodeName:}" failed. No retries permitted until 2026-10-30 21:24:47.217999323 +0000 UTC m=+1430.629538738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls") pod "maas-api-e2e-api-b-517366-84577f449d-cfsjc" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd") : secret "maas-api-e2e-api-b-517366-serving-cert" not found
Oct 30 21:24:46.714372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:46.714334    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723ff275-9ef2-426a-979e-04728732bb1e" path="/var/lib/kubelet/pods/723ff275-9ef2-426a-979e-04728732bb1e/volumes"
Oct 30 21:24:47.246057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:47.246005    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:47.249574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:47.249544    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"maas-api-e2e-api-b-517366-84577f449d-cfsjc\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:47.331783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:47.331741    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:47.665375 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:24:47.665340    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfb9445_d7c7_4a60_a505_4105787b08dd.slice/crio-6661f3cc9c18f1cd3cabc9b98365f9759e725ae7f518789e4f6f366b648ce1f5 WatchSource:0}: Error finding container 6661f3cc9c18f1cd3cabc9b98365f9759e725ae7f518789e4f6f366b648ce1f5: Status 404 returned error can't find the container with id 6661f3cc9c18f1cd3cabc9b98365f9759e725ae7f518789e4f6f366b648ce1f5
Oct 30 21:24:47.672180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:47.672149    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:24:48.140674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:48.140637    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" event={"ID":"4dfb9445-d7c7-4a60-a505-4105787b08dd","Type":"ContainerStarted","Data":"6661f3cc9c18f1cd3cabc9b98365f9759e725ae7f518789e4f6f366b648ce1f5"}
Oct 30 21:24:49.152686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:49.152643    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" event={"ID":"4dfb9445-d7c7-4a60-a505-4105787b08dd","Type":"ContainerStarted","Data":"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"}
Oct 30 21:24:49.153108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:49.152720    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:49.209048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:49.208983    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" podStartSLOduration=4.790237886 podStartE2EDuration="5.208962208s" totalImagesPullingTime="418.724322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:49.204626628 +0000 UTC m=+1432.616166065" watchObservedRunningTime="2026-10-30 21:24:49.208962208 +0000 UTC m=+1432.620501644"
Oct 30 21:24:49.887204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:49.887163    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:24:49.918867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:49.918818    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:24:51.532323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:51.532288    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:24:51.565096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:51.565056    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:24:54.887524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:54.887463    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" podUID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" containerName="payload-processing" containerID="cri-o://3cdd2f842a7c0df9bf2e752b0f55081299f4b83f820d04a28bb813d80a4ab3f2" gracePeriod=25
Oct 30 21:24:54.919398 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:54.919362    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" podUID="0283276a-024e-4243-8b0b-fd1381fb1da2" containerName="payload-pre-processing" containerID="cri-o://fa43d17599ccbc5b202b3344a92bd0f6b1d87fd690b4176c2dccba1f4a703229" gracePeriod=25
Oct 30 21:24:55.169549 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.169287    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:24:55.249205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.249167    7037 generic.go:356] "Generic (PLEG): container finished" podID="0283276a-024e-4243-8b0b-fd1381fb1da2" containerID="fa43d17599ccbc5b202b3344a92bd0f6b1d87fd690b4176c2dccba1f4a703229" exitCode=0
Oct 30 21:24:55.249378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.249329    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" event={"ID":"0283276a-024e-4243-8b0b-fd1381fb1da2","Type":"ContainerDied","Data":"fa43d17599ccbc5b202b3344a92bd0f6b1d87fd690b4176c2dccba1f4a703229"}
Oct 30 21:24:55.251428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.251401    7037 generic.go:356] "Generic (PLEG): container finished" podID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" containerID="3cdd2f842a7c0df9bf2e752b0f55081299f4b83f820d04a28bb813d80a4ab3f2" exitCode=0
Oct 30 21:24:55.251556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.251469    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" event={"ID":"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c","Type":"ContainerDied","Data":"3cdd2f842a7c0df9bf2e752b0f55081299f4b83f820d04a28bb813d80a4ab3f2"}
Oct 30 21:24:55.251556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.251527    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd" event={"ID":"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c","Type":"ContainerDied","Data":"a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d"}
Oct 30 21:24:55.251556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.251547    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3596fa3e996b6ea7a1b30bac9e2275af55502fbda065da55ba81f531c64e26d"
Oct 30 21:24:55.265306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.265274    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:24:55.325892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.325847    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:24:55.387303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387272    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\") pod \"0283276a-024e-4243-8b0b-fd1381fb1da2\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") "
Oct 30 21:24:55.387512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387359    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\" (UniqueName: \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\") pod \"0283276a-024e-4243-8b0b-fd1381fb1da2\" (UID: \"0283276a-024e-4243-8b0b-fd1381fb1da2\") "
Oct 30 21:24:55.387598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387584    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\" (UniqueName: \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\") pod \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") "
Oct 30 21:24:55.387662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\") pod \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\" (UID: \"25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c\") "
Oct 30 21:24:55.388038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387639    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume" pod "0283276a-024e-4243-8b0b-fd1381fb1da2" (UID: "0283276a-024e-4243-8b0b-fd1381fb1da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:55.388038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.387943    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume" pod "25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" (UID: "25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:55.388310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.388291    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:55.388376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.388315    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0283276a-024e-4243-8b0b-fd1381fb1da2-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:55.389684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.389663    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps" pod "25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" (UID: "25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c"). InnerVolumeSpecName "kube-api-access-bhbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:55.389946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.389933    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9" pod "0283276a-024e-4243-8b0b-fd1381fb1da2" (UID: "0283276a-024e-4243-8b0b-fd1381fb1da2"). InnerVolumeSpecName "kube-api-access-pmzv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:55.490248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.490171    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhbps\" (UniqueName: \"kubernetes.io/projected/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c-kube-api-access-bhbps\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:55.490248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:55.490200    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzv9\" (UniqueName: \"kubernetes.io/projected/0283276a-024e-4243-8b0b-fd1381fb1da2-kube-api-access-pmzv9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:56.261091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.261010    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"
Oct 30 21:24:56.261091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.261047    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm" event={"ID":"0283276a-024e-4243-8b0b-fd1381fb1da2","Type":"ContainerDied","Data":"62db3b5e52f902ac2b9fa266d33a8e032f2e3037b1afcba8af75d153ff0cd811"}
Oct 30 21:24:56.261582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.261136    7037 scope.go:122] "RemoveContainer" containerID="fa43d17599ccbc5b202b3344a92bd0f6b1d87fd690b4176c2dccba1f4a703229"
Oct 30 21:24:56.261582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.261383    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"
Oct 30 21:24:56.313548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.313520    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:24:56.325507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.325471    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4266b7-5b54d5d5f9-rflpm"]
Oct 30 21:24:56.348049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.348017    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:24:56.352600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.352574    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4266b7-6f95cbbb78-s4vfd"]
Oct 30 21:24:56.533332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.533238    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" podUID="91a38aa1-189b-41e0-9388-065575036560" containerName="payload-processing" containerID="cri-o://cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d" gracePeriod=25
Oct 30 21:24:56.566152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.566096    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" podUID="55bea80c-2b92-4d7f-b11a-803ba7f30978" containerName="payload-pre-processing" containerID="cri-o://f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd" gracePeriod=25
Oct 30 21:24:56.713854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.713820    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0283276a-024e-4243-8b0b-fd1381fb1da2" path="/var/lib/kubelet/pods/0283276a-024e-4243-8b0b-fd1381fb1da2/volumes"
Oct 30 21:24:56.714320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.714297    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" path="/var/lib/kubelet/pods/25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c/volumes"
Oct 30 21:24:56.937430 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.937397    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:24:56.939943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:56.939924    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:24:57.015811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.015776    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\") pod \"55bea80c-2b92-4d7f-b11a-803ba7f30978\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") "
Oct 30 21:24:57.015980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.015867    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\" (UniqueName: \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\") pod \"91a38aa1-189b-41e0-9388-065575036560\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") "
Oct 30 21:24:57.015980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.015931    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\") pod \"91a38aa1-189b-41e0-9388-065575036560\" (UID: \"91a38aa1-189b-41e0-9388-065575036560\") "
Oct 30 21:24:57.015980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.015951    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\" (UniqueName: \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\") pod \"55bea80c-2b92-4d7f-b11a-803ba7f30978\" (UID: \"55bea80c-2b92-4d7f-b11a-803ba7f30978\") "
Oct 30 21:24:57.016282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.016251    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume" pod "55bea80c-2b92-4d7f-b11a-803ba7f30978" (UID: "55bea80c-2b92-4d7f-b11a-803ba7f30978"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:57.016703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.016675    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume" pod "91a38aa1-189b-41e0-9388-065575036560" (UID: "91a38aa1-189b-41e0-9388-065575036560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:57.017222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.017197    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a38aa1-189b-41e0-9388-065575036560-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:57.017340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.017225    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bea80c-2b92-4d7f-b11a-803ba7f30978-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:57.025266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.018646    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr" pod "55bea80c-2b92-4d7f-b11a-803ba7f30978" (UID: "55bea80c-2b92-4d7f-b11a-803ba7f30978"). InnerVolumeSpecName "kube-api-access-gzntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:57.025266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.022002    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm" pod "91a38aa1-189b-41e0-9388-065575036560" (UID: "91a38aa1-189b-41e0-9388-065575036560"). InnerVolumeSpecName "kube-api-access-gbdnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:57.122101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.122070    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbdnm\" (UniqueName: \"kubernetes.io/projected/91a38aa1-189b-41e0-9388-065575036560-kube-api-access-gbdnm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:57.122101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.122098    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzntr\" (UniqueName: \"kubernetes.io/projected/55bea80c-2b92-4d7f-b11a-803ba7f30978-kube-api-access-gzntr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:24:57.271458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.271423    7037 generic.go:356] "Generic (PLEG): container finished" podID="55bea80c-2b92-4d7f-b11a-803ba7f30978" containerID="f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd" exitCode=0
Oct 30 21:24:57.271933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.271476    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"
Oct 30 21:24:57.271933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.271510    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" event={"ID":"55bea80c-2b92-4d7f-b11a-803ba7f30978","Type":"ContainerDied","Data":"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"}
Oct 30 21:24:57.271933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.271561    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb" event={"ID":"55bea80c-2b92-4d7f-b11a-803ba7f30978","Type":"ContainerDied","Data":"f1ece9533ad8ec65ee7384cad23d21149ead6284e0d3e8837bfd44c33975f2cd"}
Oct 30 21:24:57.271933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.271583    7037 scope.go:122] "RemoveContainer" containerID="f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"
Oct 30 21:24:57.273200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.273175    7037 generic.go:356] "Generic (PLEG): container finished" podID="91a38aa1-189b-41e0-9388-065575036560" containerID="cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d" exitCode=0
Oct 30 21:24:57.273308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.273263    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" event={"ID":"91a38aa1-189b-41e0-9388-065575036560","Type":"ContainerDied","Data":"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"}
Oct 30 21:24:57.273308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.273268    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"
Oct 30 21:24:57.273308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.273299    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj" event={"ID":"91a38aa1-189b-41e0-9388-065575036560","Type":"ContainerDied","Data":"97924ae29cf17da671aad0d23616990e793f00b1fd950577312032d0027571f4"}
Oct 30 21:24:57.285514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.285478    7037 scope.go:122] "RemoveContainer" containerID="f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"
Oct 30 21:24:57.286294 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:57.286132    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd\": container with ID starting with f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd not found: ID does not exist" containerID="f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"
Oct 30 21:24:57.286421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.286305    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd"} err="failed to get container status \"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd\": rpc error: code = NotFound desc = could not find container \"f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd\": container with ID starting with f5c652368a2a50fb2eecb08e43d53020cc3f19f5d4cb0cb46584c2cc3d8f5ccd not found: ID does not exist"
Oct 30 21:24:57.286421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.286328    7037 scope.go:122] "RemoveContainer" containerID="cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"
Oct 30 21:24:57.297321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.297301    7037 scope.go:122] "RemoveContainer" containerID="cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"
Oct 30 21:24:57.297812 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:24:57.297789    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d\": container with ID starting with cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d not found: ID does not exist" containerID="cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"
Oct 30 21:24:57.297895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.297817    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d"} err="failed to get container status \"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d\": rpc error: code = NotFound desc = could not find container \"cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d\": container with ID starting with cba50bf0c407ab5fdac54a2c09d1b7f4c822b1300fc3183079cdd7648f40830d not found: ID does not exist"
Oct 30 21:24:57.339541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.339511    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:24:57.351358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.351332    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2c7ca6-79759dcb87-qdztb"]
Oct 30 21:24:57.373984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.373959    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:24:57.380856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.380833    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2c7ca6-7cdc8bd888-bpjlj"]
Oct 30 21:24:57.850739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.850708    7037 scope.go:122] "RemoveContainer" containerID="3e35aeb1ad2c2596f65b07f1f2babcc5fa248a26c17fbf427cbe4b70bf6a2b1d"
Oct 30 21:24:57.861294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:57.861111    7037 scope.go:122] "RemoveContainer" containerID="deae3dcabb1fe4c23f9593f2a1c8db9f14703bee87b4ef7e0fe92a2853077301"
Oct 30 21:24:58.711083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:58.711042    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bea80c-2b92-4d7f-b11a-803ba7f30978" path="/var/lib/kubelet/pods/55bea80c-2b92-4d7f-b11a-803ba7f30978/volumes"
Oct 30 21:24:58.711581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:24:58.711567    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a38aa1-189b-41e0-9388-065575036560" path="/var/lib/kubelet/pods/91a38aa1-189b-41e0-9388-065575036560/volumes"
Oct 30 21:25:03.674030 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:25:03.673989    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:25:13.482776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.482739    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:25:13.483813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483782    7037 cpu_manager.go:414] "containerMap: removing container" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy"
Oct 30 21:25:13.483813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483810    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy"
Oct 30 21:25:13.484060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483904    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0283276a-024e-4243-8b0b-fd1381fb1da2" containerName="payload-pre-processing"
Oct 30 21:25:13.484060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483916    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0283276a-024e-4243-8b0b-fd1381fb1da2" containerName="payload-pre-processing"
Oct 30 21:25:13.484060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483932    7037 cpu_manager.go:414] "containerMap: removing container" podUID="55bea80c-2b92-4d7f-b11a-803ba7f30978" containerName="payload-pre-processing"
Oct 30 21:25:13.484060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.483940    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bea80c-2b92-4d7f-b11a-803ba7f30978" containerName="payload-pre-processing"
Oct 30 21:25:13.484278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484146    7037 cpu_manager.go:414] "containerMap: removing container" podUID="91a38aa1-189b-41e0-9388-065575036560" containerName="payload-processing"
Oct 30 21:25:13.484278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484161    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a38aa1-189b-41e0-9388-065575036560" containerName="payload-processing"
Oct 30 21:25:13.484278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484178    7037 cpu_manager.go:414] "containerMap: removing container" podUID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" containerName="payload-processing"
Oct 30 21:25:13.484278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484184    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" containerName="payload-processing"
Oct 30 21:25:13.484456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484377    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0283276a-024e-4243-8b0b-fd1381fb1da2" containerName="payload-pre-processing"
Oct 30 21:25:13.484456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484387    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bea80c-2b92-4d7f-b11a-803ba7f30978" containerName="payload-pre-processing"
Oct 30 21:25:13.484456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484392    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a38aa1-189b-41e0-9388-065575036560" containerName="payload-processing"
Oct 30 21:25:13.484456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484400    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c9d4b6-c5fd-4bfd-931e-596a5f37dc5c" containerName="payload-processing"
Oct 30 21:25:13.484456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.484413    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="723ff275-9ef2-426a-979e-04728732bb1e" containerName="istio-proxy"
Oct 30 21:25:13.491067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.491043    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.516102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.516075    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:25:13.521225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.521200    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1eab65\""
Oct 30 21:25:13.521335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.521239    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1eab65-dockercfg-w26ml\""
Oct 30 21:25:13.626344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.619242    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.626344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.619439    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmbp\" (UniqueName: \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.728342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.723637    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmbp\" (UniqueName: \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.728342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.723870    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.728342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.724628    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.767585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.763134    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmbp\" (UniqueName: \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\") pod \"payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:13.814628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:13.813244    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:14.132766 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:14.132736    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5def4fe2_57fc_4b77_a717_e7fd30197625.slice/crio-8f34b93318ad937205effa5da97c9ba001baf90b012389e2b24e583a5d1257e2 WatchSource:0}: Error finding container 8f34b93318ad937205effa5da97c9ba001baf90b012389e2b24e583a5d1257e2: Status 404 returned error can't find the container with id 8f34b93318ad937205effa5da97c9ba001baf90b012389e2b24e583a5d1257e2
Oct 30 21:25:14.134054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:14.134034    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:25:14.428983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:14.428885    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" event={"ID":"5def4fe2-57fc-4b77-a717-e7fd30197625","Type":"ContainerStarted","Data":"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"}
Oct 30 21:25:14.428983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:14.428949    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" event={"ID":"5def4fe2-57fc-4b77-a717-e7fd30197625","Type":"ContainerStarted","Data":"8f34b93318ad937205effa5da97c9ba001baf90b012389e2b24e583a5d1257e2"}
Oct 30 21:25:14.428983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:14.428978    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:14.473170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:14.473120    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" podStartSLOduration=1.473103646 podStartE2EDuration="1.473103646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:14.468753132 +0000 UTC m=+1457.880292580" watchObservedRunningTime="2026-10-30 21:25:14.473103646 +0000 UTC m=+1457.884643089"
Oct 30 21:25:16.883883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:16.883841    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:25:16.917066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:16.917031    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:25:21.884316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:21.884271    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" podUID="abd089c0-79b9-4b34-b6b3-f38d52751496" containerName="payload-processing" containerID="cri-o://60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb" gracePeriod=25
Oct 30 21:25:21.918280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:21.918202    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" podUID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" containerName="payload-pre-processing" containerID="cri-o://624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6" gracePeriod=25
Oct 30 21:25:22.004888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.004853    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:25:22.010991 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.010965    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.049099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.049066    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:25:22.085318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085279    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085355    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jqd\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085426    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085571    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085662    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085761    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085787    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.085835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.085826    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.101768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.101711    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c31999-openshift-default-dockercfg-j7gl5\""
Oct 30 21:25:22.187985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.187891    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.187985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.187935    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jqd\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.187985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.187970    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188078    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188133    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188203    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188221    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188245    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188394    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.188882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188852    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.189014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.188927    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.189198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.189174    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.190144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.189411    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.190144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.190102    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.192739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.192715    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.195879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.195856    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.213140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.212811    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.224153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.224125    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jqd\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-lxzhs\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.251287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.251256    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:25:22.289905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.288952    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\") pod \"abd089c0-79b9-4b34-b6b3-f38d52751496\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") "
Oct 30 21:25:22.291211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.291170    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume" pod "abd089c0-79b9-4b34-b6b3-f38d52751496" (UID: "abd089c0-79b9-4b34-b6b3-f38d52751496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:22.292113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.292089    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\" (UniqueName: \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\") pod \"abd089c0-79b9-4b34-b6b3-f38d52751496\" (UID: \"abd089c0-79b9-4b34-b6b3-f38d52751496\") "
Oct 30 21:25:22.293673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.293645    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd089c0-79b9-4b34-b6b3-f38d52751496-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:22.315203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.315176    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:25:22.317357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.317322    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf" pod "abd089c0-79b9-4b34-b6b3-f38d52751496" (UID: "abd089c0-79b9-4b34-b6b3-f38d52751496"). InnerVolumeSpecName "kube-api-access-22tmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:22.354985 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.354948    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:22.395698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.395659    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\" (UniqueName: \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\") pod \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") "
Oct 30 21:25:22.395879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.395747    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\") pod \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\" (UID: \"75579726-a0de-4f74-9ac1-f88f8ef6ad32\") "
Oct 30 21:25:22.396466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.396445    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tmf\" (UniqueName: \"kubernetes.io/projected/abd089c0-79b9-4b34-b6b3-f38d52751496-kube-api-access-22tmf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:22.396877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.396849    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume" pod "75579726-a0de-4f74-9ac1-f88f8ef6ad32" (UID: "75579726-a0de-4f74-9ac1-f88f8ef6ad32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:22.399201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.399178    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb" pod "75579726-a0de-4f74-9ac1-f88f8ef6ad32" (UID: "75579726-a0de-4f74-9ac1-f88f8ef6ad32"). InnerVolumeSpecName "kube-api-access-7jjxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:22.496259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.496222    7037 generic.go:356] "Generic (PLEG): container finished" podID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" containerID="624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6" exitCode=0
Oct 30 21:25:22.496640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.496607    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" event={"ID":"75579726-a0de-4f74-9ac1-f88f8ef6ad32","Type":"ContainerDied","Data":"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"}
Oct 30 21:25:22.496847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.496827    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww" event={"ID":"75579726-a0de-4f74-9ac1-f88f8ef6ad32","Type":"ContainerDied","Data":"8995945fab76d962def7b93f48f56d83b3db8b0461e88c28c21908be4c4bf478"}
Oct 30 21:25:22.497068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.497029    7037 scope.go:122] "RemoveContainer" containerID="624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"
Oct 30 21:25:22.497236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.497219    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"
Oct 30 21:25:22.498928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.498901    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjxb\" (UniqueName: \"kubernetes.io/projected/75579726-a0de-4f74-9ac1-f88f8ef6ad32-kube-api-access-7jjxb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:22.499067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.499053    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75579726-a0de-4f74-9ac1-f88f8ef6ad32-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:22.500290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.499989    7037 generic.go:356] "Generic (PLEG): container finished" podID="abd089c0-79b9-4b34-b6b3-f38d52751496" containerID="60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb" exitCode=0
Oct 30 21:25:22.500290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.500078    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"
Oct 30 21:25:22.500290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.500245    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" event={"ID":"abd089c0-79b9-4b34-b6b3-f38d52751496","Type":"ContainerDied","Data":"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"}
Oct 30 21:25:22.502289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.502230    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj" event={"ID":"abd089c0-79b9-4b34-b6b3-f38d52751496","Type":"ContainerDied","Data":"9f42c62b388d4feaab36fbb0d2659d7c6fbecff670f37e58e07b839720401c0f"}
Oct 30 21:25:22.526650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.526612    7037 scope.go:122] "RemoveContainer" containerID="624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"
Oct 30 21:25:22.527121 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:25:22.527089    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6\": container with ID starting with 624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6 not found: ID does not exist" containerID="624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"
Oct 30 21:25:22.527301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.527272    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6"} err="failed to get container status \"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6\": rpc error: code = NotFound desc = could not find container \"624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6\": container with ID starting with 624d33fc7353febeea944885163676c005ccf76f956861ad3d9ff725e2bfc1a6 not found: ID does not exist"
Oct 30 21:25:22.527454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.527439    7037 scope.go:122] "RemoveContainer" containerID="60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"
Oct 30 21:25:22.540263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.540240    7037 scope.go:122] "RemoveContainer" containerID="60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"
Oct 30 21:25:22.540611 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:25:22.540589    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb\": container with ID starting with 60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb not found: ID does not exist" containerID="60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"
Oct 30 21:25:22.540679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.540619    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb"} err="failed to get container status \"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb\": rpc error: code = NotFound desc = could not find container \"60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb\": container with ID starting with 60e9e9bb6d80f893c566ebb8aaa6cd1931f4a0c6f2899c087db4fa3f3f3a1aeb not found: ID does not exist"
Oct 30 21:25:22.571070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.571030    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:25:22.576272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.576241    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1da2183-8649468f78-r2ncj"]
Oct 30 21:25:22.608554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.608520    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:25:22.612397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.612372    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1da2183-677d9df459-hm4ww"]
Oct 30 21:25:22.699662 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:22.699632    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b921d44_4189_41c2_94c0_8cc915a00dde.slice/crio-f00b440703e4a354379c56e17655c9b5e69cdf644296ccfb015723c8042adc54 WatchSource:0}: Error finding container f00b440703e4a354379c56e17655c9b5e69cdf644296ccfb015723c8042adc54: Status 404 returned error can't find the container with id f00b440703e4a354379c56e17655c9b5e69cdf644296ccfb015723c8042adc54
Oct 30 21:25:22.702858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.702819    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:25:22.702956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.702889    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:25:22.702956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.702917    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:25:22.711830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.711805    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" path="/var/lib/kubelet/pods/75579726-a0de-4f74-9ac1-f88f8ef6ad32/volumes"
Oct 30 21:25:22.712382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.712365    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd089c0-79b9-4b34-b6b3-f38d52751496" path="/var/lib/kubelet/pods/abd089c0-79b9-4b34-b6b3-f38d52751496/volumes"
Oct 30 21:25:22.712900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:22.712879    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:25:23.592586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:23.592545    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" event={"ID":"6b921d44-4189-41c2-94c0-8cc915a00dde","Type":"ContainerStarted","Data":"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"}
Oct 30 21:25:23.593111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:23.592614    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" event={"ID":"6b921d44-4189-41c2-94c0-8cc915a00dde","Type":"ContainerStarted","Data":"f00b440703e4a354379c56e17655c9b5e69cdf644296ccfb015723c8042adc54"}
Oct 30 21:25:23.687777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:23.687711    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podStartSLOduration=2.687690099 podStartE2EDuration="2.687690099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:23.683872598 +0000 UTC m=+1467.095412036" watchObservedRunningTime="2026-10-30 21:25:23.687690099 +0000 UTC m=+1467.099229539"
Oct 30 21:25:24.355451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:24.355412    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:24.357571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:24.357541    7037 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 21:25:24.357742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:24.357607    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 21:25:25.360612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:25.358727    7037 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 21:25:25.360612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:25.358788    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 21:25:25.455514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:25.455104    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:25:26.358515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:26.355983    7037 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 21:25:26.358515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:26.356098    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 21:25:27.356014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:27.355978    7037 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 21:25:27.356435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:27.356029    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 21:25:28.355449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:28.355413    7037 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Oct 30 21:25:28.355741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:28.355579    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Oct 30 21:25:29.361505 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:29.361458    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:29.362019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:29.361999    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:29.362728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:29.362705    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:25:33.673864 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:25:33.673814    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:25:34.033932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.033849    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:25:34.034141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.034086    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" podUID="e671c7b3-4a88-43f1-8729-88e447654cc7" containerName="maas-api" containerID="cri-o://0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9" gracePeriod=30
Oct 30 21:25:34.380868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.380514    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:25:34.471794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.471761    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\") pod \"e671c7b3-4a88-43f1-8729-88e447654cc7\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") "
Oct 30 21:25:34.471959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.471942    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\") pod \"e671c7b3-4a88-43f1-8729-88e447654cc7\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") "
Oct 30 21:25:34.472030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.472003    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\") pod \"e671c7b3-4a88-43f1-8729-88e447654cc7\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") "
Oct 30 21:25:34.472241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.472220    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\" (UniqueName: \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\") pod \"e671c7b3-4a88-43f1-8729-88e447654cc7\" (UID: \"e671c7b3-4a88-43f1-8729-88e447654cc7\") "
Oct 30 21:25:34.472421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.472393    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca" pod "e671c7b3-4a88-43f1-8729-88e447654cc7" (UID: "e671c7b3-4a88-43f1-8729-88e447654cc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:34.472853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.472834    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e671c7b3-4a88-43f1-8729-88e447654cc7-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:34.474077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.474047    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs" pod "e671c7b3-4a88-43f1-8729-88e447654cc7" (UID: "e671c7b3-4a88-43f1-8729-88e447654cc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:34.474542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.474476    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg" pod "e671c7b3-4a88-43f1-8729-88e447654cc7" (UID: "e671c7b3-4a88-43f1-8729-88e447654cc7"). InnerVolumeSpecName "kube-api-access-mxgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:34.474673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.474626    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls" pod "e671c7b3-4a88-43f1-8729-88e447654cc7" (UID: "e671c7b3-4a88-43f1-8729-88e447654cc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:34.574284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.574248    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:34.574284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.574282    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgzg\" (UniqueName: \"kubernetes.io/projected/e671c7b3-4a88-43f1-8729-88e447654cc7-kube-api-access-mxgzg\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:34.574284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.574292    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e671c7b3-4a88-43f1-8729-88e447654cc7-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:34.779104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.778975    7037 generic.go:356] "Generic (PLEG): container finished" podID="e671c7b3-4a88-43f1-8729-88e447654cc7" containerID="0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9" exitCode=0
Oct 30 21:25:34.779994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.779939    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" event={"ID":"e671c7b3-4a88-43f1-8729-88e447654cc7","Type":"ContainerDied","Data":"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"}
Oct 30 21:25:34.780121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.780023    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8" event={"ID":"e671c7b3-4a88-43f1-8729-88e447654cc7","Type":"ContainerDied","Data":"e11c0a000fc9437fa6b9b3b915360fc59a4c665117a299cc0a64b09ae1de98b7"}
Oct 30 21:25:34.780121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.780064    7037 scope.go:122] "RemoveContainer" containerID="0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"
Oct 30 21:25:34.780416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.780378    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"
Oct 30 21:25:34.862480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.862440    7037 scope.go:122] "RemoveContainer" containerID="0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"
Oct 30 21:25:34.867400 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:25:34.865523    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9\": container with ID starting with 0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9 not found: ID does not exist" containerID="0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"
Oct 30 21:25:34.867400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.866225    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9"} err="failed to get container status \"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9\": rpc error: code = NotFound desc = could not find container \"0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9\": container with ID starting with 0cd632fe62d14806e3cc570de0520b915eaae80c822696c48813d94b711b7ff9 not found: ID does not exist"
Oct 30 21:25:34.869383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.869354    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:25:34.873392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:34.873352    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1da2183-66768659c7-vfrh8"]
Oct 30 21:25:36.709481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:36.709446    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e671c7b3-4a88-43f1-8729-88e447654cc7" path="/var/lib/kubelet/pods/e671c7b3-4a88-43f1-8729-88e447654cc7/volumes"
Oct 30 21:25:41.635314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:41.635278    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:25:41.666728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:41.666687    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:25:44.141838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.141801    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:25:44.142681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142664    7037 cpu_manager.go:414] "containerMap: removing container" podUID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" containerName="payload-pre-processing"
Oct 30 21:25:44.142755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142683    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" containerName="payload-pre-processing"
Oct 30 21:25:44.142755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142693    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e671c7b3-4a88-43f1-8729-88e447654cc7" containerName="maas-api"
Oct 30 21:25:44.142755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142698    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e671c7b3-4a88-43f1-8729-88e447654cc7" containerName="maas-api"
Oct 30 21:25:44.142755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142743    7037 cpu_manager.go:414] "containerMap: removing container" podUID="abd089c0-79b9-4b34-b6b3-f38d52751496" containerName="payload-processing"
Oct 30 21:25:44.142755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142752    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd089c0-79b9-4b34-b6b3-f38d52751496" containerName="payload-processing"
Oct 30 21:25:44.142905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142896    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="75579726-a0de-4f74-9ac1-f88f8ef6ad32" containerName="payload-pre-processing"
Oct 30 21:25:44.142939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142907    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e671c7b3-4a88-43f1-8729-88e447654cc7" containerName="maas-api"
Oct 30 21:25:44.142939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.142925    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd089c0-79b9-4b34-b6b3-f38d52751496" containerName="payload-processing"
Oct 30 21:25:44.148220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.148202    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.169276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.169244    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:25:44.177132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.177104    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-79c24a\""
Oct 30 21:25:44.177325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.177174    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-79c24a-dockercfg-xrsn7\""
Oct 30 21:25:44.195427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.195386    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.195652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.195636    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwrg\" (UniqueName: \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.198813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.198789    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:25:44.203721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.203703    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.229137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.227993    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:25:44.313513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.297036    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2mf\" (UniqueName: \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.313513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.297292    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwrg\" (UniqueName: \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.313513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.297358    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.313513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.297451    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.323601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.318003    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.357758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.354801    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwrg\" (UniqueName: \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\") pod \"payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.418632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.418399    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.418830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.418664    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2mf\" (UniqueName: \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.419262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.419242    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.439724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.439594    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2mf\" (UniqueName: \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\") pod \"payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.463428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.463389    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:44.527941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.527903    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:44.818597 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:44.818563    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49eb04a7_c42c_4eb7_9f58_a94d1e345e2c.slice/crio-505dbc219d235757e2898b6d5ab5def5e8cc5a6f06e8380959f863a3af90d310 WatchSource:0}: Error finding container 505dbc219d235757e2898b6d5ab5def5e8cc5a6f06e8380959f863a3af90d310: Status 404 returned error can't find the container with id 505dbc219d235757e2898b6d5ab5def5e8cc5a6f06e8380959f863a3af90d310
Oct 30 21:25:44.819707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.819445    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:25:44.869037 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:44.868996    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766bc989_145d_42c9_b2d9_4916f69186c6.slice/crio-bcd68c534cbad84645f0b94be11751c6411a1adbe5a92ad0f54ab10799b05f91 WatchSource:0}: Error finding container bcd68c534cbad84645f0b94be11751c6411a1adbe5a92ad0f54ab10799b05f91: Status 404 returned error can't find the container with id bcd68c534cbad84645f0b94be11751c6411a1adbe5a92ad0f54ab10799b05f91
Oct 30 21:25:44.870795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.870770    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:25:44.876004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:44.875980    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" event={"ID":"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c","Type":"ContainerStarted","Data":"505dbc219d235757e2898b6d5ab5def5e8cc5a6f06e8380959f863a3af90d310"}
Oct 30 21:25:45.885001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.884954    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" event={"ID":"766bc989-145d-42c9-b2d9-4916f69186c6","Type":"ContainerStarted","Data":"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"}
Oct 30 21:25:45.885511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.885062    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:45.885511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.885090    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" event={"ID":"766bc989-145d-42c9-b2d9-4916f69186c6","Type":"ContainerStarted","Data":"bcd68c534cbad84645f0b94be11751c6411a1adbe5a92ad0f54ab10799b05f91"}
Oct 30 21:25:45.886736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.886710    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" event={"ID":"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c","Type":"ContainerStarted","Data":"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"}
Oct 30 21:25:45.886858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.886846    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:45.929288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.929238    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" podStartSLOduration=1.929225809 podStartE2EDuration="1.929225809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:45.925762144 +0000 UTC m=+1489.337301593" watchObservedRunningTime="2026-10-30 21:25:45.929225809 +0000 UTC m=+1489.340765299"
Oct 30 21:25:45.965581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:45.965511    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" podStartSLOduration=1.9654803790000002 podStartE2EDuration="1.965480379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:45.964107617 +0000 UTC m=+1489.375647058" watchObservedRunningTime="2026-10-30 21:25:45.965480379 +0000 UTC m=+1489.377019818"
Oct 30 21:25:46.636583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.636459    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" podUID="30f9dde6-6e58-4656-a033-356592b117fc" containerName="payload-processing" containerID="cri-o://aea6ac697a4e211ca368c7e109b7e1644c1d24a67381857a7684195abdfee5fc" gracePeriod=25
Oct 30 21:25:46.667326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.667293    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" podUID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" containerName="payload-pre-processing" containerID="cri-o://c9fd386a3c3c206ad41592a3918434b9793d6e36839f7c690765583edf5b67c7" gracePeriod=25
Oct 30 21:25:46.905721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.905627    7037 generic.go:356] "Generic (PLEG): container finished" podID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" containerID="c9fd386a3c3c206ad41592a3918434b9793d6e36839f7c690765583edf5b67c7" exitCode=0
Oct 30 21:25:46.906182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.905758    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" event={"ID":"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d","Type":"ContainerDied","Data":"c9fd386a3c3c206ad41592a3918434b9793d6e36839f7c690765583edf5b67c7"}
Oct 30 21:25:46.907744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.907721    7037 generic.go:356] "Generic (PLEG): container finished" podID="30f9dde6-6e58-4656-a033-356592b117fc" containerID="aea6ac697a4e211ca368c7e109b7e1644c1d24a67381857a7684195abdfee5fc" exitCode=0
Oct 30 21:25:46.908665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:46.908641    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" event={"ID":"30f9dde6-6e58-4656-a033-356592b117fc","Type":"ContainerDied","Data":"aea6ac697a4e211ca368c7e109b7e1644c1d24a67381857a7684195abdfee5fc"}
Oct 30 21:25:47.028733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.028707    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:25:47.033232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.033210    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:25:47.191062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.190956    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\" (UniqueName: \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\") pod \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") "
Oct 30 21:25:47.191062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.191028    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\") pod \"30f9dde6-6e58-4656-a033-356592b117fc\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") "
Oct 30 21:25:47.191305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.191102    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\") pod \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\" (UID: \"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d\") "
Oct 30 21:25:47.191369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.191354    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\" (UniqueName: \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\") pod \"30f9dde6-6e58-4656-a033-356592b117fc\" (UID: \"30f9dde6-6e58-4656-a033-356592b117fc\") "
Oct 30 21:25:47.192814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.192784    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume" pod "30f9dde6-6e58-4656-a033-356592b117fc" (UID: "30f9dde6-6e58-4656-a033-356592b117fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:47.192954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.192783    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume" pod "7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" (UID: "7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:47.194733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.194705    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q" pod "7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" (UID: "7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d"). InnerVolumeSpecName "kube-api-access-njp5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:47.194858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.194734    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr" pod "30f9dde6-6e58-4656-a033-356592b117fc" (UID: "30f9dde6-6e58-4656-a033-356592b117fc"). InnerVolumeSpecName "kube-api-access-bnjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:47.293706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.293665    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njp5q\" (UniqueName: \"kubernetes.io/projected/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-kube-api-access-njp5q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:47.293706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.293705    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f9dde6-6e58-4656-a033-356592b117fc-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:47.294162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.293722    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:47.294162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.293736    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjvr\" (UniqueName: \"kubernetes.io/projected/30f9dde6-6e58-4656-a033-356592b117fc-kube-api-access-bnjvr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:25:47.918296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.918084    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"
Oct 30 21:25:47.918296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.918232    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq" event={"ID":"7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d","Type":"ContainerDied","Data":"ff4faf56be7f8c4edfec407bf35f9d59b8be3df7ae7069a028de004da9881d88"}
Oct 30 21:25:47.918904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.918320    7037 scope.go:122] "RemoveContainer" containerID="c9fd386a3c3c206ad41592a3918434b9793d6e36839f7c690765583edf5b67c7"
Oct 30 21:25:47.920015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.919989    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"
Oct 30 21:25:47.920134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.920059    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9" event={"ID":"30f9dde6-6e58-4656-a033-356592b117fc","Type":"ContainerDied","Data":"3558c8fd366d95a2041b243295f02028db15516d7d638d8e4210016a57e735e7"}
Oct 30 21:25:47.931556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.931534    7037 scope.go:122] "RemoveContainer" containerID="aea6ac697a4e211ca368c7e109b7e1644c1d24a67381857a7684195abdfee5fc"
Oct 30 21:25:47.933155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.933133    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:25:47.934063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934047    7037 cpu_manager.go:414] "containerMap: removing container" podUID="30f9dde6-6e58-4656-a033-356592b117fc" containerName="payload-processing"
Oct 30 21:25:47.934117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934065    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f9dde6-6e58-4656-a033-356592b117fc" containerName="payload-processing"
Oct 30 21:25:47.934154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934119    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" containerName="payload-pre-processing"
Oct 30 21:25:47.934154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934125    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" containerName="payload-pre-processing"
Oct 30 21:25:47.934331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934322    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f9dde6-6e58-4656-a033-356592b117fc" containerName="payload-processing"
Oct 30 21:25:47.934366 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.934336    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" containerName="payload-pre-processing"
Oct 30 21:25:47.939994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.939975    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:47.955910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.955887    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1eab65-serving-cert\""
Oct 30 21:25:47.960916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.960894    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:25:47.986222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.986191    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:25:47.992062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:47.992025    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca18b3-668696fdfd-9w2s9"]
Oct 30 21:25:48.021347 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.021318    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:25:48.025192 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.025169    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca18b3-5c8bc94c59-twwlq"]
Oct 30 21:25:48.107799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.107769    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.107973 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.107954    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.108021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.107981    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zmr\" (UniqueName: \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.108138 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.108126    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.209341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.209245    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.209537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.209423    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.209537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.209457    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zmr\" (UniqueName: \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.209678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.209637    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.210295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.210267    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.212031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.212006    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.212235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.212215    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.232567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.230738    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zmr\" (UniqueName: \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\") pod \"maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.257548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.257506    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:48.579351 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:48.579324    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda050fd78_0321_42a2_8f58_41c0eb50c70b.slice/crio-bcdcef20aa7852d62264571ddd72b59c91bbc98116861dcab40ce0ae6a1f0bde WatchSource:0}: Error finding container bcdcef20aa7852d62264571ddd72b59c91bbc98116861dcab40ce0ae6a1f0bde: Status 404 returned error can't find the container with id bcdcef20aa7852d62264571ddd72b59c91bbc98116861dcab40ce0ae6a1f0bde
Oct 30 21:25:48.581029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.581001    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:25:48.716789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.716754    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f9dde6-6e58-4656-a033-356592b117fc" path="/var/lib/kubelet/pods/30f9dde6-6e58-4656-a033-356592b117fc/volumes"
Oct 30 21:25:48.717329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.717310    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d" path="/var/lib/kubelet/pods/7420e6dd-23a0-4fe5-b6d8-58a4d8b22f7d/volumes"
Oct 30 21:25:48.957466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:48.957379    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" event={"ID":"a050fd78-0321-42a2-8f58-41c0eb50c70b","Type":"ContainerStarted","Data":"bcdcef20aa7852d62264571ddd72b59c91bbc98116861dcab40ce0ae6a1f0bde"}
Oct 30 21:25:49.968649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:49.968615    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" event={"ID":"a050fd78-0321-42a2-8f58-41c0eb50c70b","Type":"ContainerStarted","Data":"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"}
Oct 30 21:25:49.969066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:49.968701    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:50.014183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:50.014134    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" podStartSLOduration=2.577462567 podStartE2EDuration="3.014120607s" totalImagesPullingTime="436.65804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:50.010176089 +0000 UTC m=+1493.421715526" watchObservedRunningTime="2026-10-30 21:25:50.014120607 +0000 UTC m=+1493.425660044"
Oct 30 21:25:55.594169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.594137    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:25:55.599193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.599175    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.619073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.619042    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-093daa-serving-cert\""
Oct 30 21:25:55.625018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.624985    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:25:55.675085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.675033    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.675245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.675167    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.675245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.675206    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tpf\" (UniqueName: \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.675403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.675382    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.777861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.777816    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.778392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.778338    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.778619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.778414    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.778619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.778520    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tpf\" (UniqueName: \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.778772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.778668    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.781424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.781397    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.781715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.781695    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.799265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.799233    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tpf\" (UniqueName: \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\") pod \"maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.916352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.916273    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:55.984744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:55.984718    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:25:56.222353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:56.222327    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:25:56.223382 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:25:56.223353    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fe1de9_7a76_4845_bf18_96528a9df3d1.slice/crio-c2a739f1ee285dab60de670e7ce45cc520b0182b9e0a8e9c950ab193dae000f0 WatchSource:0}: Error finding container c2a739f1ee285dab60de670e7ce45cc520b0182b9e0a8e9c950ab193dae000f0: Status 404 returned error can't find the container with id c2a739f1ee285dab60de670e7ce45cc520b0182b9e0a8e9c950ab193dae000f0
Oct 30 21:25:56.911880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:56.911849    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:25:56.912245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:56.911932    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:25:57.037875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:57.037772    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" event={"ID":"d4fe1de9-7a76-4845-bf18-96528a9df3d1","Type":"ContainerStarted","Data":"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"}
Oct 30 21:25:57.037875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:57.037857    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" event={"ID":"d4fe1de9-7a76-4845-bf18-96528a9df3d1","Type":"ContainerStarted","Data":"c2a739f1ee285dab60de670e7ce45cc520b0182b9e0a8e9c950ab193dae000f0"}
Oct 30 21:25:57.039578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:57.038238    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:25:57.086837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:57.086776    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" podStartSLOduration=1.5912384579999999 podStartE2EDuration="2.086757103s" totalImagesPullingTime="495.518645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:57.083716959 +0000 UTC m=+1500.495256399" watchObservedRunningTime="2026-10-30 21:25:57.086757103 +0000 UTC m=+1500.498296541"
Oct 30 21:25:57.920928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:25:57.920898    7037 scope.go:122] "RemoveContainer" containerID="110e68f3c25097cd4b03a58843d1a0aff63fa5ac7a828fab6ef3e096cb175b60"
Oct 30 21:26:00.262163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.262120    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:26:00.262742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.262412    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerName="maas-api" containerID="cri-o://1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804" gracePeriod=30
Oct 30 21:26:00.425510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.425449    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.160:8443/health\": dial tcp 10.134.0.160:8443: connect: connection refused"
Oct 30 21:26:00.586240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.586204    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:26:00.599162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.599123    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:26:00.599398 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.599367    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" podUID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" containerName="maas-api" containerID="cri-o://148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4" gracePeriod=30
Oct 30 21:26:00.688623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.688574    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\") pod \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") "
Oct 30 21:26:00.689136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.689105    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\") pod \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") "
Oct 30 21:26:00.689247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.689175    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\" (UniqueName: \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\") pod \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") "
Oct 30 21:26:00.689247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.689239    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\") pod \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\" (UID: \"92ca7c2a-c7dc-4fa0-a1cc-094643011b26\") "
Oct 30 21:26:00.691153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.690929    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca" pod "92ca7c2a-c7dc-4fa0-a1cc-094643011b26" (UID: "92ca7c2a-c7dc-4fa0-a1cc-094643011b26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:00.697157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.692718    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs" pod "92ca7c2a-c7dc-4fa0-a1cc-094643011b26" (UID: "92ca7c2a-c7dc-4fa0-a1cc-094643011b26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:00.697157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.696397    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k" pod "92ca7c2a-c7dc-4fa0-a1cc-094643011b26" (UID: "92ca7c2a-c7dc-4fa0-a1cc-094643011b26"). InnerVolumeSpecName "kube-api-access-p7s5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:00.697586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.697558    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls" pod "92ca7c2a-c7dc-4fa0-a1cc-094643011b26" (UID: "92ca7c2a-c7dc-4fa0-a1cc-094643011b26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:00.793102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.793056    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.793102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.793091    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.793279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.793110    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7s5k\" (UniqueName: \"kubernetes.io/projected/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-kube-api-access-p7s5k\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.793279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.793125    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ca7c2a-c7dc-4fa0-a1cc-094643011b26-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.893161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.893129    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.996639    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\" (UniqueName: \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\") pod \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") "
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.996736    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\") pod \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") "
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.997077    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") pod \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") "
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.997133    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\") pod \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\" (UID: \"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a\") "
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:00.998174    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca" pod "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" (UID: "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:01.008729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.008656    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk" pod "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" (UID: "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a"). InnerVolumeSpecName "kube-api-access-dqxpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:01.012659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.012621    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs" pod "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" (UID: "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:01.031575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.017608    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls" pod "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" (UID: "f9b50a7a-ad6d-461c-894b-5c1c3ef3954a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:01.098289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.098251    7037 generic.go:356] "Generic (PLEG): container finished" podID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerID="1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804" exitCode=0
Oct 30 21:26:01.098640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.098610    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" event={"ID":"92ca7c2a-c7dc-4fa0-a1cc-094643011b26","Type":"ContainerDied","Data":"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"}
Oct 30 21:26:01.098846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.098826    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk" event={"ID":"92ca7c2a-c7dc-4fa0-a1cc-094643011b26","Type":"ContainerDied","Data":"cb736c02cacef440ad0502fb3e1448795b70ff3e485054d6b0ff48b815a18445"}
Oct 30 21:26:01.099021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.098999    7037 scope.go:122] "RemoveContainer" containerID="1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"
Oct 30 21:26:01.099289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.099273    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"
Oct 30 21:26:01.102410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.100611    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:01.102410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.100639    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:01.102410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.100651    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:01.102410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.100666    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxpk\" (UniqueName: \"kubernetes.io/projected/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a-kube-api-access-dqxpk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:01.105718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.105689    7037 generic.go:356] "Generic (PLEG): container finished" podID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" containerID="148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4" exitCode=0
Oct 30 21:26:01.105827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.105744    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" event={"ID":"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a","Type":"ContainerDied","Data":"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"}
Oct 30 21:26:01.105827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.105783    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259" event={"ID":"f9b50a7a-ad6d-461c-894b-5c1c3ef3954a","Type":"ContainerDied","Data":"5cee4329bab81a166926e69319b985971a5d693293b6b136dd4e3f0b36a6c7b2"}
Oct 30 21:26:01.105940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.105854    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"
Oct 30 21:26:01.125150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.125121    7037 scope.go:122] "RemoveContainer" containerID="1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"
Oct 30 21:26:01.125637 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:01.125610    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804\": container with ID starting with 1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804 not found: ID does not exist" containerID="1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"
Oct 30 21:26:01.125847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.125649    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804"} err="failed to get container status \"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804\": rpc error: code = NotFound desc = could not find container \"1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804\": container with ID starting with 1c4ce07773ce7cf68234ec45a1d31634d3235a8f11952b9c7920573de146e804 not found: ID does not exist"
Oct 30 21:26:01.125847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.125675    7037 scope.go:122] "RemoveContainer" containerID="148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"
Oct 30 21:26:01.154708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.153449    7037 scope.go:122] "RemoveContainer" containerID="148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"
Oct 30 21:26:01.154708 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:01.154025    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4\": container with ID starting with 148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4 not found: ID does not exist" containerID="148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"
Oct 30 21:26:01.154708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.154114    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4"} err="failed to get container status \"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4\": rpc error: code = NotFound desc = could not find container \"148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4\": container with ID starting with 148f0a5597e1234fd5169a4c58f091e38bebc61b80572400a9fff466f439a3e4 not found: ID does not exist"
Oct 30 21:26:01.166511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.162125    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:26:01.196063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.196030    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4266b7-6c7bb44cb7-5pfdk"]
Oct 30 21:26:01.245576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.245460    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:26:01.256423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:01.256391    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2c7ca6-7bd75f7c56-29259"]
Oct 30 21:26:02.709720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:02.709689    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" path="/var/lib/kubelet/pods/92ca7c2a-c7dc-4fa0-a1cc-094643011b26/volumes"
Oct 30 21:26:02.710172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:02.710157    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" path="/var/lib/kubelet/pods/f9b50a7a-ad6d-461c-894b-5c1c3ef3954a/volumes"
Oct 30 21:26:03.050993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:03.050918    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:26:03.673938 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:03.673899    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:26:15.664322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.664278    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:26:15.665190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665171    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" containerName="maas-api"
Oct 30 21:26:15.665190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665190    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" containerName="maas-api"
Oct 30 21:26:15.665276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665217    7037 cpu_manager.go:414] "containerMap: removing container" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerName="maas-api"
Oct 30 21:26:15.665276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665225    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerName="maas-api"
Oct 30 21:26:15.665369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665359    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ca7c2a-c7dc-4fa0-a1cc-094643011b26" containerName="maas-api"
Oct 30 21:26:15.665402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.665380    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b50a7a-ad6d-461c-894b-5c1c3ef3954a" containerName="maas-api"
Oct 30 21:26:15.669258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.669239    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.679395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.679366    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfkm\" (UniqueName: \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.679547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.679475    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.690276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.690236    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:26:15.697417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.697378    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-79e911\""
Oct 30 21:26:15.733528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.733481    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:26:15.738785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.738761    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.759059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.759024    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:26:15.760377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.760349    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-79e911-dockercfg-g6p9c\""
Oct 30 21:26:15.781736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.781688    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.781940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.781820    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvbt\" (UniqueName: \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.782057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.782028    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfkm\" (UniqueName: \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.782130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.782099    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.784256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.784209    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.879044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.876513    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfkm\" (UniqueName: \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\") pod \"payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:15.883728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.883685    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.883999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.883975    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvbt\" (UniqueName: \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.885690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.885663    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.907649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.907613    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvbt\" (UniqueName: \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\") pod \"payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:15.983812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:15.983732    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:16.065858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:16.065821    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:16.315276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:16.315199    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:26:16.321683 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:26:16.321649    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f74923_7a74_40b9_862d_758eabfcd3e2.slice/crio-cb2e40a0f1ee239dbdbd8fccbec1cfafd09e023d5e3b207c7b1b095feff4cdc0 WatchSource:0}: Error finding container cb2e40a0f1ee239dbdbd8fccbec1cfafd09e023d5e3b207c7b1b095feff4cdc0: Status 404 returned error can't find the container with id cb2e40a0f1ee239dbdbd8fccbec1cfafd09e023d5e3b207c7b1b095feff4cdc0
Oct 30 21:26:16.439681 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:26:16.439644    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdacc84e_96aa_4e22_bb76_d96f130a80c0.slice/crio-2addec4ab468166c22326cc721d12b6056dd93544b4028584fccacfa807d033b WatchSource:0}: Error finding container 2addec4ab468166c22326cc721d12b6056dd93544b4028584fccacfa807d033b: Status 404 returned error can't find the container with id 2addec4ab468166c22326cc721d12b6056dd93544b4028584fccacfa807d033b
Oct 30 21:26:16.441887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:16.441849    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:26:17.272852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.272812    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" event={"ID":"07f74923-7a74-40b9-862d-758eabfcd3e2","Type":"ContainerStarted","Data":"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"}
Oct 30 21:26:17.273897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.272880    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" event={"ID":"07f74923-7a74-40b9-862d-758eabfcd3e2","Type":"ContainerStarted","Data":"cb2e40a0f1ee239dbdbd8fccbec1cfafd09e023d5e3b207c7b1b095feff4cdc0"}
Oct 30 21:26:17.273897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.272912    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:17.274893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.274856    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" event={"ID":"cdacc84e-96aa-4e22-bb76-d96f130a80c0","Type":"ContainerStarted","Data":"1d79344c44bb8eb79751dd6066b0b13011fce3a82f819c80b2f78aa7beb603d6"}
Oct 30 21:26:17.275044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.274918    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" event={"ID":"cdacc84e-96aa-4e22-bb76-d96f130a80c0","Type":"ContainerStarted","Data":"2addec4ab468166c22326cc721d12b6056dd93544b4028584fccacfa807d033b"}
Oct 30 21:26:17.275044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.274949    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:17.341950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.341898    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" podStartSLOduration=2.341878772 podStartE2EDuration="2.341878772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:17.337830697 +0000 UTC m=+1520.749370133" watchObservedRunningTime="2026-10-30 21:26:17.341878772 +0000 UTC m=+1520.753418209"
Oct 30 21:26:17.357202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.357166    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:26:17.362599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.362571    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.381594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.381540    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-79c24a-serving-cert\""
Oct 30 21:26:17.381594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.381566    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:26:17.391867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.391814    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" podStartSLOduration=2.39179566 podStartE2EDuration="2.39179566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:17.389376464 +0000 UTC m=+1520.800915903" watchObservedRunningTime="2026-10-30 21:26:17.39179566 +0000 UTC m=+1520.803335096"
Oct 30 21:26:17.409372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.409333    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.409698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.409469    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.409698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.409563    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.409873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.409850    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkdl\" (UniqueName: \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.511591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.511549    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkdl\" (UniqueName: \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.511797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.511691    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.511797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.511757    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.511797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.511780    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.512641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.512611    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.514402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.514381    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.514554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.514441    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.536107 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.536024    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkdl\" (UniqueName: \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\") pod \"maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:17.689068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:17.689029    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:18.062511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:18.062456    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:26:18.286899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:18.286862    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" event={"ID":"d15cc2c4-7970-49ec-93b9-e514ee7b51bc","Type":"ContainerStarted","Data":"7cfe6268da2719c4efe4c6c4c643e9c43d083f333a2b870f57e3e3528909260e"}
Oct 30 21:26:19.295567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:19.295530    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" event={"ID":"d15cc2c4-7970-49ec-93b9-e514ee7b51bc","Type":"ContainerStarted","Data":"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"}
Oct 30 21:26:19.296055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:19.295610    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:19.341080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:19.341031    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" podStartSLOduration=2.019450972 podStartE2EDuration="2.341017043s" totalImagesPullingTime="321.566071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:19.339015478 +0000 UTC m=+1522.750554917" watchObservedRunningTime="2026-10-30 21:26:19.341017043 +0000 UTC m=+1522.752556479"
Oct 30 21:26:25.309150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:25.309122    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:26:28.289970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:28.289939    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:26:28.290344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:28.290015    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:26:33.673456 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:33.673417    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:26:46.587455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:46.587375    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:26:46.587993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:46.587616    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy" containerID="cri-o://320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41" gracePeriod=30
Oct 30 21:26:46.655599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:46.655563    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:26:46.655849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:46.655821    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy" containerID="cri-o://3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e" gracePeriod=30
Oct 30 21:26:47.965960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:47.965923    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:26:47.966782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:47.966728    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy" containerID="cri-o://2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831" gracePeriod=30
Oct 30 21:26:52.004773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.004746    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:26:52.043114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.043090    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:26:52.110133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110092    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.110336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.110336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110240    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110268    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110299    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110355    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110394    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110474    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110512    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110555    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110604    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110671    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110745    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.110936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110795    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.110936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110841    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:52.110936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110880    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.111222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.110982    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.111222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111098    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.111222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111123    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111133    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\") pod \"1f733c03-8ae2-485c-90a3-e8d96a82753c\" (UID: \"1f733c03-8ae2-485c-90a3-e8d96a82753c\") "
Oct 30 21:26:52.111447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111240    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\") pod \"47beaf0a-34f3-4571-9b21-32c7083b2d67\" (UID: \"47beaf0a-34f3-4571-9b21-32c7083b2d67\") "
Oct 30 21:26:52.111447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111249    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111332    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111378    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:52.111707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111514    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111535    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111819    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.111961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.111871    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115220    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f733c03-8ae2-485c-90a3-e8d96a82753c-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115258    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115274    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115335    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115353    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115370    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115383    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115407    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115421    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115435    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47beaf0a-34f3-4571-9b21-32c7083b2d67-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115899    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.115984    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.116005    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.116036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "kube-api-access-pjrl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.116104    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "kube-api-access-2vsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:52.125514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.116596    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:52.131163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.127677    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo" pod "47beaf0a-34f3-4571-9b21-32c7083b2d67" (UID: "47beaf0a-34f3-4571-9b21-32c7083b2d67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:52.131163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.128471    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy" pod "1f733c03-8ae2-485c-90a3-e8d96a82753c" (UID: "1f733c03-8ae2-485c-90a3-e8d96a82753c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:52.217540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217453    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217482    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217509    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217523    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vsz4\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-kube-api-access-2vsz4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217538    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrl2\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-kube-api-access-pjrl2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217552    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217564    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47beaf0a-34f3-4571-9b21-32c7083b2d67-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.217811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.217573    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f733c03-8ae2-485c-90a3-e8d96a82753c-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:52.591893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.591861    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:26:52.592738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592721    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy"
Oct 30 21:26:52.592801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592739    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy"
Oct 30 21:26:52.592801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592793    7037 cpu_manager.go:414] "containerMap: removing container" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy"
Oct 30 21:26:52.592801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592799    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy"
Oct 30 21:26:52.592924 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592915    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerName="istio-proxy"
Oct 30 21:26:52.592960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.592934    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerName="istio-proxy"
Oct 30 21:26:52.598010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.597990    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.618361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.618294    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1eab65-openshift-default-dockercfg-5jsjw\""
Oct 30 21:26:52.643634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.643607    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:26:52.657853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.657821    7037 generic.go:356] "Generic (PLEG): container finished" podID="1f733c03-8ae2-485c-90a3-e8d96a82753c" containerID="3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e" exitCode=0
Oct 30 21:26:52.658010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.657883    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"
Oct 30 21:26:52.658010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.657900    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" event={"ID":"1f733c03-8ae2-485c-90a3-e8d96a82753c","Type":"ContainerDied","Data":"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"}
Oct 30 21:26:52.658010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.657958    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4" event={"ID":"1f733c03-8ae2-485c-90a3-e8d96a82753c","Type":"ContainerDied","Data":"eca18e63d3d946cbf22df3451f510067f77e1ea549aa23bd229a633db5b84214"}
Oct 30 21:26:52.658010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.657990    7037 scope.go:122] "RemoveContainer" containerID="3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"
Oct 30 21:26:52.660021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.659994    7037 generic.go:356] "Generic (PLEG): container finished" podID="47beaf0a-34f3-4571-9b21-32c7083b2d67" containerID="320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41" exitCode=0
Oct 30 21:26:52.660153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.660060    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"
Oct 30 21:26:52.660153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.660118    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" event={"ID":"47beaf0a-34f3-4571-9b21-32c7083b2d67","Type":"ContainerDied","Data":"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"}
Oct 30 21:26:52.660273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.660159    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22" event={"ID":"47beaf0a-34f3-4571-9b21-32c7083b2d67","Type":"ContainerDied","Data":"8d03eac0f0448c6812e6437070b9b5c8791512b0630064f08d31ffb46b883725"}
Oct 30 21:26:52.671983 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.671964    7037 scope.go:122] "RemoveContainer" containerID="3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"
Oct 30 21:26:52.672442 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:52.672383    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e\": container with ID starting with 3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e not found: ID does not exist" containerID="3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"
Oct 30 21:26:52.672620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.672454    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e"} err="failed to get container status \"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e\": rpc error: code = NotFound desc = could not find container \"3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e\": container with ID starting with 3a36826b25ded48783dec6fa63166c9cd198c4a621905eaee0c01c79f26a024e not found: ID does not exist"
Oct 30 21:26:52.672620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.672478    7037 scope.go:122] "RemoveContainer" containerID="320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"
Oct 30 21:26:52.684558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.684537    7037 scope.go:122] "RemoveContainer" containerID="320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"
Oct 30 21:26:52.684871 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:52.684844    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41\": container with ID starting with 320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41 not found: ID does not exist" containerID="320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"
Oct 30 21:26:52.684943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.684882    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41"} err="failed to get container status \"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41\": rpc error: code = NotFound desc = could not find container \"320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41\": container with ID starting with 320c8aee4de7044cc1bb517202495394f4faf752e8145e7c067374cbf2272b41 not found: ID does not exist"
Oct 30 21:26:52.720794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.720763    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:26:52.728365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.728341    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-k5dq4"]
Oct 30 21:26:52.744708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.744680    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxlzd\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.744866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.744726    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.744919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.744884    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.744963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.744926    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.745018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.745004    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.745124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.745110    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.745171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.745148    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.745228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.745171    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.745387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.745363    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.764477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.764437    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:26:52.769097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.769075    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-8bt22"]
Oct 30 21:26:52.846448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846372    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846567    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxlzd\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846619    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846743    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846779    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846812    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.846885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846815    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.847053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.846956    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.847053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.847022    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.847157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.847069    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.847428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.847403    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.847563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.847484    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.848134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.848103    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.848242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.848194    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.849961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.849931    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.850478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.850451    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.886629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.886606    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.888048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.888024    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxlzd\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:52.956840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:52.956799    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:53.254271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.254239    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:26:53.289582 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:26:53.289478    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c297a8b_24b5_4179_a5ba_3b5ec90c851f.slice/crio-193c42fa839b5ad5e34a997d921994cc654065bae1f46672f377a95f587b04f7 WatchSource:0}: Error finding container 193c42fa839b5ad5e34a997d921994cc654065bae1f46672f377a95f587b04f7: Status 404 returned error can't find the container with id 193c42fa839b5ad5e34a997d921994cc654065bae1f46672f377a95f587b04f7
Oct 30 21:26:53.290559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.290364    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:26:53.293730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.293693    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:53.293841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.293776    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:53.293841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.293822    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:53.356511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.356048    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.356754    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.356808    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.356840    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.356928    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.357020    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.357186    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.357229    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.357294    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\") pod \"8ef6aea4-d75b-4622-8020-81773b7a77d5\" (UID: \"8ef6aea4-d75b-4622-8020-81773b7a77d5\") "
Oct 30 21:26:53.357834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.357576    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:53.358604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358393    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:53.358693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358629    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:53.358978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358794    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:53.358978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358966    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.359115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358982    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.359115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.358994    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.359115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.359006    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.359479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.359443    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:53.360684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.360652    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:53.363399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.363371    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:26:53.364761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.364738    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.364761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.364760    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.364916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.364806    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.364916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.364814    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.365030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.365021    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.365087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.365039    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerName="istio-proxy"
Oct 30 21:26:53.365654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.365627    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:53.365759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.365672    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:53.367615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.367518    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr" pod "8ef6aea4-d75b-4622-8020-81773b7a77d5" (UID: "8ef6aea4-d75b-4622-8020-81773b7a77d5"). InnerVolumeSpecName "kube-api-access-gp4xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:53.372074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.372053    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.400925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.400894    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:26:53.410361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.410288    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-79c24a-openshift-default-dockercfg-k6ggn\""
Oct 30 21:26:53.461025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.460986    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461174    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461216    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461270    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461420    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461597    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461663    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461714    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461738    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hth\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.461884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461871    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4xr\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-kube-api-access-gp4xr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.461958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461891    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.461958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461906    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef6aea4-d75b-4622-8020-81773b7a77d5-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.461958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461921    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.461958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.461934    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef6aea4-d75b-4622-8020-81773b7a77d5-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.563275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563238    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563306    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563359    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563382    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hth\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563447    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563577    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563611    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563664    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.563865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.563818    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.564607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.564577    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.564844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.564778    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.565320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.565272    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.565448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.565428    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.565623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.565562    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.567537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.567476    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.568032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.568011    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.671655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.641109    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.671655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.641739    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hth\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.734342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.733651    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:53.804545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.804470    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" event={"ID":"9c297a8b-24b5-4179-a5ba-3b5ec90c851f","Type":"ContainerStarted","Data":"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"}
Oct 30 21:26:53.804796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.804777    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" event={"ID":"9c297a8b-24b5-4179-a5ba-3b5ec90c851f","Type":"ContainerStarted","Data":"193c42fa839b5ad5e34a997d921994cc654065bae1f46672f377a95f587b04f7"}
Oct 30 21:26:53.811577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.811182    7037 generic.go:356] "Generic (PLEG): container finished" podID="8ef6aea4-d75b-4622-8020-81773b7a77d5" containerID="2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831" exitCode=0
Oct 30 21:26:53.811577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.811267    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerDied","Data":"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"}
Oct 30 21:26:53.811577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.811314    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92" event={"ID":"8ef6aea4-d75b-4622-8020-81773b7a77d5","Type":"ContainerDied","Data":"bdfef0319d4f6cd5732f186be1cc62ca32d2c2e72afa060a323814e2a27e6ce3"}
Oct 30 21:26:53.811577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.811348    7037 scope.go:122] "RemoveContainer" containerID="2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"
Oct 30 21:26:53.817546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.817462    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"
Oct 30 21:26:53.865736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.860653    7037 scope.go:122] "RemoveContainer" containerID="a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"
Oct 30 21:26:53.873736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.873677    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podStartSLOduration=1.873638209 podStartE2EDuration="1.873638209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:53.872596763 +0000 UTC m=+1557.284136201" watchObservedRunningTime="2026-10-30 21:26:53.873638209 +0000 UTC m=+1557.285177646"
Oct 30 21:26:53.898599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.898545    7037 scope.go:122] "RemoveContainer" containerID="2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"
Oct 30 21:26:53.899017 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:53.898980    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831\": container with ID starting with 2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831 not found: ID does not exist" containerID="2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"
Oct 30 21:26:53.899102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.899026    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831"} err="failed to get container status \"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831\": rpc error: code = NotFound desc = could not find container \"2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831\": container with ID starting with 2d52f4c923ca92e1f393ad2c9f97fa7be70c8ac907342d964a53e6517c325831 not found: ID does not exist"
Oct 30 21:26:53.899102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.899050    7037 scope.go:122] "RemoveContainer" containerID="a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"
Oct 30 21:26:53.899451 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:26:53.899428    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e\": container with ID starting with a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e not found: ID does not exist" containerID="a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"
Oct 30 21:26:53.899618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.899460    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e"} err="failed to get container status \"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e\": rpc error: code = NotFound desc = could not find container \"a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e\": container with ID starting with a7766efac31ea318d4dac4e320ef12482b78a129d291a269c59abe487a0a381e not found: ID does not exist"
Oct 30 21:26:53.945866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.945834    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:26:53.949784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.949757    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-pzv92"]
Oct 30 21:26:53.957095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:53.957073    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:26:54.192723 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:26:54.192685    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9354139_0450_47ac_a4f1_92906cd69734.slice/crio-0c5fb2f8987e3872f914df0191014facca8a4a197bd980e93bb0294987a3f923 WatchSource:0}: Error finding container 0c5fb2f8987e3872f914df0191014facca8a4a197bd980e93bb0294987a3f923: Status 404 returned error can't find the container with id 0c5fb2f8987e3872f914df0191014facca8a4a197bd980e93bb0294987a3f923
Oct 30 21:26:54.193545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.193509    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:26:54.196350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.196308    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:54.196425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.196403    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:54.196464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.196451    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:26:54.756762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.756718    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f733c03-8ae2-485c-90a3-e8d96a82753c" path="/var/lib/kubelet/pods/1f733c03-8ae2-485c-90a3-e8d96a82753c/volumes"
Oct 30 21:26:54.758097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.758064    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47beaf0a-34f3-4571-9b21-32c7083b2d67" path="/var/lib/kubelet/pods/47beaf0a-34f3-4571-9b21-32c7083b2d67/volumes"
Oct 30 21:26:54.758730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.758709    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef6aea4-d75b-4622-8020-81773b7a77d5" path="/var/lib/kubelet/pods/8ef6aea4-d75b-4622-8020-81773b7a77d5/volumes"
Oct 30 21:26:54.860589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.860549    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" event={"ID":"d9354139-0450-47ac-a4f1-92906cd69734","Type":"ContainerStarted","Data":"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"}
Oct 30 21:26:54.860781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.860614    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" event={"ID":"d9354139-0450-47ac-a4f1-92906cd69734","Type":"ContainerStarted","Data":"0c5fb2f8987e3872f914df0191014facca8a4a197bd980e93bb0294987a3f923"}
Oct 30 21:26:54.906124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.906070    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podStartSLOduration=1.906056112 podStartE2EDuration="1.906056112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:54.905024001 +0000 UTC m=+1558.316563439" watchObservedRunningTime="2026-10-30 21:26:54.906056112 +0000 UTC m=+1558.317595548"
Oct 30 21:26:54.958917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.958881    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:54.959133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:54.958967    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:26:55.742932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:55.742894    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:26:55.747629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:55.747596    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:26:55.747883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:55.747837    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:26:55.989306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:55.987333    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:55.989306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:55.988755    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:26:56.752516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:56.751980    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:26:56.752516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:56.752051    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:26:56.964513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:56.963892    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:56.964513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:56.963971    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:26:57.762517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:57.762253    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:26:57.762517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:57.762314    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:26:57.959516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:57.958816    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:57.959516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:57.958880    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:26:58.681154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.681123    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:26:58.686329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.686303    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:26:58.776584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.757035    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:26:58.776584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.757092    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:26:58.996795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.995540    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:58.996795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:58.995610    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:26:59.752515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:59.748861    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:26:59.752515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:59.748921    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:26:59.962397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:59.959428    7037 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:26:59.962397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:26:59.959504    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 21:27:00.748610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:00.746700    7037 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 30 21:27:00.748610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:00.746758    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 30 21:27:00.963054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:00.963019    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:27:00.963543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:00.963523    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:27:00.964189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:00.964167    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:27:01.748531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:01.746471    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:27:01.748531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:01.746920    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:27:01.748531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:01.747680    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:27:03.674307 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:27:03.674270    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:27:03.681445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.681410    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" podUID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" containerName="payload-processing" containerID="cri-o://00369b47fa3c84b0ab76d0951e839fd216c360b08889a8a0358ca02ea74838c1" gracePeriod=25
Oct 30 21:27:03.686949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.686914    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" podUID="942b1948-a3d8-4538-903b-c7ce9179dcfb" containerName="payload-pre-processing" containerID="cri-o://6c0d3c01da2ce04d57b1955027e6368b01b1532239e111fd3426f64397b5afe4" gracePeriod=25
Oct 30 21:27:03.992203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.992143    7037 generic.go:356] "Generic (PLEG): container finished" podID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" containerID="00369b47fa3c84b0ab76d0951e839fd216c360b08889a8a0358ca02ea74838c1" exitCode=0
Oct 30 21:27:03.992364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.992343    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" event={"ID":"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11","Type":"ContainerDied","Data":"00369b47fa3c84b0ab76d0951e839fd216c360b08889a8a0358ca02ea74838c1"}
Oct 30 21:27:03.995031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.994767    7037 generic.go:356] "Generic (PLEG): container finished" podID="942b1948-a3d8-4538-903b-c7ce9179dcfb" containerID="6c0d3c01da2ce04d57b1955027e6368b01b1532239e111fd3426f64397b5afe4" exitCode=0
Oct 30 21:27:03.995031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:03.994968    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" event={"ID":"942b1948-a3d8-4538-903b-c7ce9179dcfb","Type":"ContainerDied","Data":"6c0d3c01da2ce04d57b1955027e6368b01b1532239e111fd3426f64397b5afe4"}
Oct 30 21:27:04.054227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.054197    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:27:04.096408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.096377    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:27:04.128261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.128223    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\") pod \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") "
Oct 30 21:27:04.128667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.128590    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\" (UniqueName: \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\") pod \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\" (UID: \"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11\") "
Oct 30 21:27:04.128667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.128633    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume" pod "2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" (UID: "2f5706d3-f5ac-44bd-902d-e78fd5ba3e11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:04.129478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.129420    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:04.131159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.131130    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw" pod "2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" (UID: "2f5706d3-f5ac-44bd-902d-e78fd5ba3e11"). InnerVolumeSpecName "kube-api-access-svdbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:04.231229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.231181    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\" (UniqueName: \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\") pod \"942b1948-a3d8-4538-903b-c7ce9179dcfb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") "
Oct 30 21:27:04.231485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.231371    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\") pod \"942b1948-a3d8-4538-903b-c7ce9179dcfb\" (UID: \"942b1948-a3d8-4538-903b-c7ce9179dcfb\") "
Oct 30 21:27:04.231904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.231873    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume" pod "942b1948-a3d8-4538-903b-c7ce9179dcfb" (UID: "942b1948-a3d8-4538-903b-c7ce9179dcfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:04.233153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.233088    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942b1948-a3d8-4538-903b-c7ce9179dcfb-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:04.233153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.233116    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdbw\" (UniqueName: \"kubernetes.io/projected/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11-kube-api-access-svdbw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:04.233652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.233615    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk" pod "942b1948-a3d8-4538-903b-c7ce9179dcfb" (UID: "942b1948-a3d8-4538-903b-c7ce9179dcfb"). InnerVolumeSpecName "kube-api-access-ftnpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:04.334641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:04.334604    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnpk\" (UniqueName: \"kubernetes.io/projected/942b1948-a3d8-4538-903b-c7ce9179dcfb-kube-api-access-ftnpk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:05.016008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.015901    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb" event={"ID":"942b1948-a3d8-4538-903b-c7ce9179dcfb","Type":"ContainerDied","Data":"f0ccc9e31223c274c11f97e6e1f503f9bfdf0b545eb8f7cba02cf1ad4c3c9c4d"}
Oct 30 21:27:05.017300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.016462    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"
Oct 30 21:27:05.017535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.017510    7037 scope.go:122] "RemoveContainer" containerID="6c0d3c01da2ce04d57b1955027e6368b01b1532239e111fd3426f64397b5afe4"
Oct 30 21:27:05.026597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.025300    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x" event={"ID":"2f5706d3-f5ac-44bd-902d-e78fd5ba3e11","Type":"ContainerDied","Data":"971313627c0d266f235259d6fb206e7d8bfefa68c64e6b190d7743f3b7c886a0"}
Oct 30 21:27:05.026597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.025436    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"
Oct 30 21:27:05.040970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.040946    7037 scope.go:122] "RemoveContainer" containerID="00369b47fa3c84b0ab76d0951e839fd216c360b08889a8a0358ca02ea74838c1"
Oct 30 21:27:05.082656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.082622    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:27:05.094306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.094274    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517366-86d679464c-rh6sb"]
Oct 30 21:27:05.138603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.138565    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:27:05.150144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:05.150113    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517366-7c5b497f8d-nrz2x"]
Oct 30 21:27:06.710177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:06.710144    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" path="/var/lib/kubelet/pods/2f5706d3-f5ac-44bd-902d-e78fd5ba3e11/volumes"
Oct 30 21:27:06.710672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:06.710656    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942b1948-a3d8-4538-903b-c7ce9179dcfb" path="/var/lib/kubelet/pods/942b1948-a3d8-4538-903b-c7ce9179dcfb/volumes"
Oct 30 21:27:16.712195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.712162    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:27:16.712937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.712921    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" containerName="payload-processing"
Oct 30 21:27:16.712984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.712939    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" containerName="payload-processing"
Oct 30 21:27:16.712984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.712980    7037 cpu_manager.go:414] "containerMap: removing container" podUID="942b1948-a3d8-4538-903b-c7ce9179dcfb" containerName="payload-pre-processing"
Oct 30 21:27:16.713045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.712989    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942b1948-a3d8-4538-903b-c7ce9179dcfb" containerName="payload-pre-processing"
Oct 30 21:27:16.713219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.713206    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5706d3-f5ac-44bd-902d-e78fd5ba3e11" containerName="payload-processing"
Oct 30 21:27:16.713285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.713229    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="942b1948-a3d8-4538-903b-c7ce9179dcfb" containerName="payload-pre-processing"
Oct 30 21:27:16.722779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.722753    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.744447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.744410    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:27:16.751543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.751518    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3b28bc\""
Oct 30 21:27:16.838634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.838600    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:27:16.847761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.847726    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:16.871555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.871521    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3b28bc-dockercfg-zpbgv\""
Oct 30 21:27:16.873661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.873618    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.873827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.873802    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rxr\" (UniqueName: \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.874334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.874310    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:27:16.976818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.976174    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:16.976818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.976627    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnvq\" (UniqueName: \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:16.976818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.976751    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.977116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.976881    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rxr\" (UniqueName: \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.979265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.978101    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:16.998736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:16.998689    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rxr\" (UniqueName: \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\") pod \"payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:17.039093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.039059    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:17.080324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.078833    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:17.080324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.079009    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnvq\" (UniqueName: \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:17.080324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.080272    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:17.100377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.100348    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnvq\" (UniqueName: \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\") pod \"payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:17.163941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.163903    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:17.447414 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:27:17.447376    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1983658c_79f1_41da_bb35_435d236894fa.slice/crio-c9db57c040edc72a3cf905a0d525024883bc7889f21496cde17c44a8b8b419c9 WatchSource:0}: Error finding container c9db57c040edc72a3cf905a0d525024883bc7889f21496cde17c44a8b8b419c9: Status 404 returned error can't find the container with id c9db57c040edc72a3cf905a0d525024883bc7889f21496cde17c44a8b8b419c9
Oct 30 21:27:17.460990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.460908    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:27:17.554227 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:27:17.554200    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00697119_a608_4d04_9dca_ab40504da067.slice/crio-71ef79a13e1e06b93db50984be7149f573d10b763ca8fc0a08669ebd345c2a35 WatchSource:0}: Error finding container 71ef79a13e1e06b93db50984be7149f573d10b763ca8fc0a08669ebd345c2a35: Status 404 returned error can't find the container with id 71ef79a13e1e06b93db50984be7149f573d10b763ca8fc0a08669ebd345c2a35
Oct 30 21:27:17.556277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:17.556247    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:27:18.144481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.144442    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" event={"ID":"1983658c-79f1-41da-bb35-435d236894fa","Type":"ContainerStarted","Data":"6eb8ac5f6cf8d4aa6a5063d8af27d74761b7c6b6b3009ab6e5db4a30df1917b7"}
Oct 30 21:27:18.145011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.144546    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" event={"ID":"1983658c-79f1-41da-bb35-435d236894fa","Type":"ContainerStarted","Data":"c9db57c040edc72a3cf905a0d525024883bc7889f21496cde17c44a8b8b419c9"}
Oct 30 21:27:18.145011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.144571    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:18.146452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.146428    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" event={"ID":"00697119-a608-4d04-9dca-ab40504da067","Type":"ContainerStarted","Data":"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"}
Oct 30 21:27:18.146565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.146471    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" event={"ID":"00697119-a608-4d04-9dca-ab40504da067","Type":"ContainerStarted","Data":"71ef79a13e1e06b93db50984be7149f573d10b763ca8fc0a08669ebd345c2a35"}
Oct 30 21:27:18.146565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.146527    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:18.198394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.198327    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" podStartSLOduration=2.198307489 podStartE2EDuration="2.198307489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:18.195538018 +0000 UTC m=+1581.607077452" watchObservedRunningTime="2026-10-30 21:27:18.198307489 +0000 UTC m=+1581.609846928"
Oct 30 21:27:18.252036 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:18.251966    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" podStartSLOduration=2.251947367 podStartE2EDuration="2.251947367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:18.247543527 +0000 UTC m=+1581.659082964" watchObservedRunningTime="2026-10-30 21:27:18.251947367 +0000 UTC m=+1581.663486804"
Oct 30 21:27:22.385898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.385858    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:27:22.390715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.390691    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.417852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.417801    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:27:22.423299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.423273    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-290ede-serving-cert\""
Oct 30 21:27:22.479018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.478983    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.479215 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.479187    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.479290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.479248    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rh7w\" (UniqueName: \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.479424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.479408    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.581273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.581240    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.581463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.581365    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.581463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.581400    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rh7w\" (UniqueName: \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.581606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.581518    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.582127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.582102    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.583784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.583761    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.583934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.583914    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.621196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.610053    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rh7w\" (UniqueName: \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\") pod \"maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:22.713824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:22.713747    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:23.055761 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:27:23.055733    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cceade5_5911_42e1_8149_1c5839f2c4a0.slice/crio-dad0c5b3d998167761a4651eb0d53c6fb18e04f9d9d3d9f7c8d3b00892292f52 WatchSource:0}: Error finding container dad0c5b3d998167761a4651eb0d53c6fb18e04f9d9d3d9f7c8d3b00892292f52: Status 404 returned error can't find the container with id dad0c5b3d998167761a4651eb0d53c6fb18e04f9d9d3d9f7c8d3b00892292f52
Oct 30 21:27:23.057700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:23.057677    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:27:23.057797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:23.057710    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:27:23.193105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:23.193062    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" event={"ID":"3cceade5-5911-42e1-8149-1c5839f2c4a0","Type":"ContainerStarted","Data":"dad0c5b3d998167761a4651eb0d53c6fb18e04f9d9d3d9f7c8d3b00892292f52"}
Oct 30 21:27:24.201922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:24.201874    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" event={"ID":"3cceade5-5911-42e1-8149-1c5839f2c4a0","Type":"ContainerStarted","Data":"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"}
Oct 30 21:27:24.202349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:24.201975    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:24.250358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:24.250306    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" podStartSLOduration=1.875395241 podStartE2EDuration="2.250292126s" totalImagesPullingTime="374.896885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:24.245241667 +0000 UTC m=+1587.656781101" watchObservedRunningTime="2026-10-30 21:27:24.250292126 +0000 UTC m=+1587.661831639"
Oct 30 21:27:29.158173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:29.158141    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:27:29.158675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:29.158214    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:27:30.217153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:30.217123    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:27:33.674067 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:27:33.674030    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:27:37.640907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.640869    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:27:37.646574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.646544    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.667931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.667896    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b2628a-serving-cert\""
Oct 30 21:27:37.688901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.688867    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:27:37.737277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.737234    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxp4\" (UniqueName: \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.737480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.737298    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.737480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.737406    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.737654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.737507    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.841771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.841732    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxp4\" (UniqueName: \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.841976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.841790    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.841976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.841836    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.841976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.841894    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.843346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.843320    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.844986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.844957    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.845413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.845391    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.867652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.867513    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxp4\" (UniqueName: \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\") pod \"maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:37.970083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:37.969957    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:38.286373 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:27:38.286342    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9a0bbc_0ff0_4580_aa3a_ce01dcc160f2.slice/crio-3abb9f30e97fb6b6a5f5efa1e5a55b6cd04d3e973ef95dc8bd0d35e645311a61 WatchSource:0}: Error finding container 3abb9f30e97fb6b6a5f5efa1e5a55b6cd04d3e973ef95dc8bd0d35e645311a61: Status 404 returned error can't find the container with id 3abb9f30e97fb6b6a5f5efa1e5a55b6cd04d3e973ef95dc8bd0d35e645311a61
Oct 30 21:27:38.289114 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:38.289092    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:27:38.321928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:38.321893    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" event={"ID":"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2","Type":"ContainerStarted","Data":"3abb9f30e97fb6b6a5f5efa1e5a55b6cd04d3e973ef95dc8bd0d35e645311a61"}
Oct 30 21:27:39.333038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:39.333003    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" event={"ID":"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2","Type":"ContainerStarted","Data":"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"}
Oct 30 21:27:39.333584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:39.333085    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:39.381526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:39.381463    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" podStartSLOduration=2.073998649 podStartE2EDuration="2.38144911s" totalImagesPullingTime="307.450461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:39.379864535 +0000 UTC m=+1602.791403975" watchObservedRunningTime="2026-10-30 21:27:39.38144911 +0000 UTC m=+1602.792988547"
Oct 30 21:27:45.346687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:45.346653    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:27:46.729583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:46.729549    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:27:46.730065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:46.729783    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy" containerID="cri-o://4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e" gracePeriod=30
Oct 30 21:27:48.152124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:48.152084    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:27:48.152565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:48.152326    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy" containerID="cri-o://e25ebd1542cb8f44c87f25343e7659f36f8c1c7649a8cc6d0311a9b51bb2c641" gracePeriod=30
Oct 30 21:27:52.038408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.038379    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:27:52.137188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.137134    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.137515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.137476    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.137687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.137670    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.138112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138091    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.138219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138123    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138332    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138515    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138539    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138578    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138706    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\") pod \"30262c1c-c6bc-49a7-b1cc-238910051a8b\" (UID: \"30262c1c-c6bc-49a7-b1cc-238910051a8b\") "
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.138801    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139017    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139184    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:52.139581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139331    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:52.140111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139775    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.140111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139797    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30262c1c-c6bc-49a7-b1cc-238910051a8b-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.140111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139813    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.140111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139826    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.140111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.139838    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.142394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.142361    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:52.142546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.142476    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:27:52.143907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.143883    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:52.151673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.151637    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2" pod "30262c1c-c6bc-49a7-b1cc-238910051a8b" (UID: "30262c1c-c6bc-49a7-b1cc-238910051a8b"). InnerVolumeSpecName "kube-api-access-c92j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:52.242412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.242368    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92j2\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-kube-api-access-c92j2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.242412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.242412    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.242729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.242427    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.242729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.242440    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30262c1c-c6bc-49a7-b1cc-238910051a8b-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:52.455918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.455883    7037 generic.go:356] "Generic (PLEG): container finished" podID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerID="4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e" exitCode=0
Oct 30 21:27:52.456124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.455935    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"
Oct 30 21:27:52.456124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.455978    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" event={"ID":"30262c1c-c6bc-49a7-b1cc-238910051a8b","Type":"ContainerDied","Data":"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"}
Oct 30 21:27:52.456124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.456046    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w" event={"ID":"30262c1c-c6bc-49a7-b1cc-238910051a8b","Type":"ContainerDied","Data":"5ea480b32993b1502585bd33cb71502c296b4d45bb17ee939cecd7c1f88f433a"}
Oct 30 21:27:52.456124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.456074    7037 scope.go:122] "RemoveContainer" containerID="4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"
Oct 30 21:27:52.478306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.478273    7037 scope.go:122] "RemoveContainer" containerID="4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"
Oct 30 21:27:52.479115 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:27:52.479092    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e\": container with ID starting with 4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e not found: ID does not exist" containerID="4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"
Oct 30 21:27:52.479207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.479129    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e"} err="failed to get container status \"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e\": rpc error: code = NotFound desc = could not find container \"4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e\": container with ID starting with 4260297fb39bf50068c915bbba957aed677d1a64d3fee5cd984b0725774ff25e not found: ID does not exist"
Oct 30 21:27:52.552751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.552639    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:27:52.566298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.566263    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-rln8w"]
Oct 30 21:27:52.713126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:52.713042    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" path="/var/lib/kubelet/pods/30262c1c-c6bc-49a7-b1cc-238910051a8b/volumes"
Oct 30 21:27:53.514639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.514546    7037 generic.go:356] "Generic (PLEG): container finished" podID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerID="e25ebd1542cb8f44c87f25343e7659f36f8c1c7649a8cc6d0311a9b51bb2c641" exitCode=0
Oct 30 21:27:53.515156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.515133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" event={"ID":"98e23218-42fc-48c9-8b42-b3e56ce080f9","Type":"ContainerDied","Data":"e25ebd1542cb8f44c87f25343e7659f36f8c1c7649a8cc6d0311a9b51bb2c641"}
Oct 30 21:27:53.605515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.597393    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:27:53.717657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717622    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.717834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717700    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.717834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717723    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.717834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717769    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.717834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717800    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.717834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717833    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.718129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717929    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.718129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717965    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:53.718129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717985    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:53.718129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.717995    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.718341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.718152    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:53.718425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.718233    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:53.718425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.718258    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\") pod \"98e23218-42fc-48c9-8b42-b3e56ce080f9\" (UID: \"98e23218-42fc-48c9-8b42-b3e56ce080f9\") "
Oct 30 21:27:53.718624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.718585    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:53.719338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.719313    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.719423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.719344    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.719423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.719372    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.719423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.719387    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.719423 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.719399    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e23218-42fc-48c9-8b42-b3e56ce080f9-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.722127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.721725    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:27:53.722127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.721807    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "kube-api-access-vhzc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:53.729645 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.729614    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:53.731046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.731017    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token" pod "98e23218-42fc-48c9-8b42-b3e56ce080f9" (UID: "98e23218-42fc-48c9-8b42-b3e56ce080f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:53.843513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.840747    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.843513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.840784    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.843513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.840798    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e23218-42fc-48c9-8b42-b3e56ce080f9-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:53.843513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:53.840812    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzc4\" (UniqueName: \"kubernetes.io/projected/98e23218-42fc-48c9-8b42-b3e56ce080f9-kube-api-access-vhzc4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:27:54.524941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.524905    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g" event={"ID":"98e23218-42fc-48c9-8b42-b3e56ce080f9","Type":"ContainerDied","Data":"57589285c0d92a7ef6dc1af4d2fc1fb9891fb56390f248d0ff0c618914d62c5b"}
Oct 30 21:27:54.525346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.524967    7037 scope.go:122] "RemoveContainer" containerID="e25ebd1542cb8f44c87f25343e7659f36f8c1c7649a8cc6d0311a9b51bb2c641"
Oct 30 21:27:54.525346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.525075    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"
Oct 30 21:27:54.639545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.639464    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:27:54.648373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.648338    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-phj9g"]
Oct 30 21:27:54.713778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:27:54.713746    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" path="/var/lib/kubelet/pods/98e23218-42fc-48c9-8b42-b3e56ce080f9/volumes"
Oct 30 21:28:03.690675 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:28:03.690637    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:28:30.143072 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.143014    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:28:30.143579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.143347    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerName="maas-api" containerID="cri-o://ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e" gracePeriod=30
Oct 30 21:28:30.163644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.163607    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.176:8443/health\": dial tcp 10.134.0.176:8443: connect: connection refused"
Oct 30 21:28:30.437945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.437915    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:28:30.628027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.627989    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") pod \"4dfb9445-d7c7-4a60-a505-4105787b08dd\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") "
Oct 30 21:28:30.628243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.628161    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\" (UniqueName: \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\") pod \"4dfb9445-d7c7-4a60-a505-4105787b08dd\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") "
Oct 30 21:28:30.628243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.628236    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\") pod \"4dfb9445-d7c7-4a60-a505-4105787b08dd\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") "
Oct 30 21:28:30.628405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.628383    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\") pod \"4dfb9445-d7c7-4a60-a505-4105787b08dd\" (UID: \"4dfb9445-d7c7-4a60-a505-4105787b08dd\") "
Oct 30 21:28:30.629568 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.629540    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca" pod "4dfb9445-d7c7-4a60-a505-4105787b08dd" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:30.631327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.631292    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4" pod "4dfb9445-d7c7-4a60-a505-4105787b08dd" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd"). InnerVolumeSpecName "kube-api-access-clql4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:30.631439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.631372    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls" pod "4dfb9445-d7c7-4a60-a505-4105787b08dd" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:30.631766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.631744    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs" pod "4dfb9445-d7c7-4a60-a505-4105787b08dd" (UID: "4dfb9445-d7c7-4a60-a505-4105787b08dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:30.730338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.730310    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:28:30.730338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.730339    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clql4\" (UniqueName: \"kubernetes.io/projected/4dfb9445-d7c7-4a60-a505-4105787b08dd-kube-api-access-clql4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:28:30.730523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.730349    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dfb9445-d7c7-4a60-a505-4105787b08dd-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:28:30.730523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.730358    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfb9445-d7c7-4a60-a505-4105787b08dd-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:28:30.870382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.870345    7037 generic.go:356] "Generic (PLEG): container finished" podID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerID="ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e" exitCode=0
Oct 30 21:28:30.870621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.870408    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"
Oct 30 21:28:30.870710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.870613    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" event={"ID":"4dfb9445-d7c7-4a60-a505-4105787b08dd","Type":"ContainerDied","Data":"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"}
Oct 30 21:28:30.870710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.870663    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc" event={"ID":"4dfb9445-d7c7-4a60-a505-4105787b08dd","Type":"ContainerDied","Data":"6661f3cc9c18f1cd3cabc9b98365f9759e725ae7f518789e4f6f366b648ce1f5"}
Oct 30 21:28:30.870710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.870690    7037 scope.go:122] "RemoveContainer" containerID="ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"
Oct 30 21:28:30.886901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.886880    7037 scope.go:122] "RemoveContainer" containerID="ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"
Oct 30 21:28:30.887292 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:28:30.887261    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e\": container with ID starting with ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e not found: ID does not exist" containerID="ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"
Oct 30 21:28:30.887386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.887301    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e"} err="failed to get container status \"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e\": rpc error: code = NotFound desc = could not find container \"ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e\": container with ID starting with ade815c281d9aa1d8745787ef2d15457bae94670f2ea4ac9e812fdc3c10fb71e not found: ID does not exist"
Oct 30 21:28:30.932543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.932459    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:28:30.940933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:30.940894    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517366-84577f449d-cfsjc"]
Oct 30 21:28:32.710040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:32.709998    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" path="/var/lib/kubelet/pods/4dfb9445-d7c7-4a60-a505-4105787b08dd/volumes"
Oct 30 21:28:33.673181 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:28:33.673143    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:28:48.587610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.587568    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:28:48.590240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590210    7037 cpu_manager.go:414] "containerMap: removing container" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerName="maas-api"
Oct 30 21:28:48.590407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590393    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerName="maas-api"
Oct 30 21:28:48.590613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590599    7037 cpu_manager.go:414] "containerMap: removing container" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy"
Oct 30 21:28:48.590717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590705    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy"
Oct 30 21:28:48.590813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590804    7037 cpu_manager.go:414] "containerMap: removing container" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy"
Oct 30 21:28:48.590888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.590878    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy"
Oct 30 21:28:48.591167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.591152    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e23218-42fc-48c9-8b42-b3e56ce080f9" containerName="istio-proxy"
Oct 30 21:28:48.591265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.591254    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="30262c1c-c6bc-49a7-b1cc-238910051a8b" containerName="istio-proxy"
Oct 30 21:28:48.591354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.591345    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfb9445-d7c7-4a60-a505-4105787b08dd" containerName="maas-api"
Oct 30 21:28:48.599071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.599046    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:28:48.603191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.599303    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.608579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.608284    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.647303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.647271    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-abf04b\""
Oct 30 21:28:48.654630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.654568    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:28:48.655752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.655728    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-abf04b-dockercfg-cfrn6\""
Oct 30 21:28:48.699246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.699203    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:28:48.758951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.758908    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.761208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.761170    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nj7\" (UniqueName: \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.761523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.761477    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.761763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.761743    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9kf8\" (UniqueName: \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.864883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.864790    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.865061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.864914    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nj7\" (UniqueName: \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.865061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.865001    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.865200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.865069    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9kf8\" (UniqueName: \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.866148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.866107    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.866296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.866278    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.923765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.923732    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nj7\" (UniqueName: \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\") pod \"payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:48.925092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.925064    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9kf8\" (UniqueName: \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\") pod \"payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.963258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.963221    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:48.978374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:48.978332    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:49.319825 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:28:49.319797    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6f6946_ba4c_4800_a521_911fae1f33e1.slice/crio-01feab42614afe7fd97d2bbedf2146e0d68305e1d0373dd066a8cb05d471da77 WatchSource:0}: Error finding container 01feab42614afe7fd97d2bbedf2146e0d68305e1d0373dd066a8cb05d471da77: Status 404 returned error can't find the container with id 01feab42614afe7fd97d2bbedf2146e0d68305e1d0373dd066a8cb05d471da77
Oct 30 21:28:49.320251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:49.320184    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:28:49.329415 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:28:49.329390    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884587fa_e1c5_4747_bf30_33f4392b6c6f.slice/crio-9d27ba4d77ff1fa9d293c5445e2217a876178e94c7f9ad14fda8a8d6ecb68666 WatchSource:0}: Error finding container 9d27ba4d77ff1fa9d293c5445e2217a876178e94c7f9ad14fda8a8d6ecb68666: Status 404 returned error can't find the container with id 9d27ba4d77ff1fa9d293c5445e2217a876178e94c7f9ad14fda8a8d6ecb68666
Oct 30 21:28:49.333551 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:49.333524    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:28:50.053704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.053669    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" event={"ID":"884587fa-e1c5-4747-bf30-33f4392b6c6f","Type":"ContainerStarted","Data":"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"}
Oct 30 21:28:50.054242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.053732    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" event={"ID":"884587fa-e1c5-4747-bf30-33f4392b6c6f","Type":"ContainerStarted","Data":"9d27ba4d77ff1fa9d293c5445e2217a876178e94c7f9ad14fda8a8d6ecb68666"}
Oct 30 21:28:50.054242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.053769    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:28:50.055704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.055672    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" event={"ID":"cc6f6946-ba4c-4800-a521-911fae1f33e1","Type":"ContainerStarted","Data":"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"}
Oct 30 21:28:50.055815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.055733    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:28:50.055815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.055759    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" event={"ID":"cc6f6946-ba4c-4800-a521-911fae1f33e1","Type":"ContainerStarted","Data":"01feab42614afe7fd97d2bbedf2146e0d68305e1d0373dd066a8cb05d471da77"}
Oct 30 21:28:50.108458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.108404    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" podStartSLOduration=2.108390444 podStartE2EDuration="2.108390444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:50.106338056 +0000 UTC m=+1673.517877495" watchObservedRunningTime="2026-10-30 21:28:50.108390444 +0000 UTC m=+1673.519929881"
Oct 30 21:28:50.169984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:50.169933    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" podStartSLOduration=2.16991892 podStartE2EDuration="2.16991892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:50.165128928 +0000 UTC m=+1673.576668365" watchObservedRunningTime="2026-10-30 21:28:50.16991892 +0000 UTC m=+1673.581458356"
Oct 30 21:28:52.858513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:52.836841    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:28:52.858513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:52.856825    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:52.895344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:52.893582    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-67f8e0\""
Oct 30 21:28:52.895344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:52.893678    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-67f8e0-dockercfg-fq9lg\""
Oct 30 21:28:52.917944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:52.917170    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:28:53.048112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.048074    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.048303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.048240    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmrt\" (UniqueName: \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.150039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.149954    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmrt\" (UniqueName: \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.150228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.150215    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.150927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.150906    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.202336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.202299    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmrt\" (UniqueName: \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\") pod \"payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.501910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.501824    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:53.866158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:53.866128    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:28:53.866537 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:28:53.866450    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad902be3_5438_4eec_9944_2038a4f1ce19.slice/crio-6442bd4c01e6c6c3e138b923036cc58587c7215b1de63292d09b16899083b602 WatchSource:0}: Error finding container 6442bd4c01e6c6c3e138b923036cc58587c7215b1de63292d09b16899083b602: Status 404 returned error can't find the container with id 6442bd4c01e6c6c3e138b923036cc58587c7215b1de63292d09b16899083b602
Oct 30 21:28:54.102857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:54.102809    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" event={"ID":"ad902be3-5438-4eec-9944-2038a4f1ce19","Type":"ContainerStarted","Data":"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"}
Oct 30 21:28:54.103058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:54.102888    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:28:54.103058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:54.102913    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" event={"ID":"ad902be3-5438-4eec-9944-2038a4f1ce19","Type":"ContainerStarted","Data":"6442bd4c01e6c6c3e138b923036cc58587c7215b1de63292d09b16899083b602"}
Oct 30 21:28:54.163944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:54.163833    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" podStartSLOduration=2.163817157 podStartE2EDuration="2.163817157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:54.161474222 +0000 UTC m=+1677.573013662" watchObservedRunningTime="2026-10-30 21:28:54.163817157 +0000 UTC m=+1677.575356594"
Oct 30 21:28:58.131938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:28:58.131905    7037 scope.go:122] "RemoveContainer" containerID="3cdd2f842a7c0df9bf2e752b0f55081299f4b83f820d04a28bb813d80a4ab3f2"
Oct 30 21:29:01.072100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:01.072063    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:29:01.072587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:01.072540    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:29:03.673792 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:29:03.673741    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:29:04.323861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:04.323819    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:29:04.324111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:04.324084    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy" containerID="cri-o://dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a" gracePeriod=30
Oct 30 21:29:05.118227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:05.118190    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:29:09.607854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.607792    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:29:09.629109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629081    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629150    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629177    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629240    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629353    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629397    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629539    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.629577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629566    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629568    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:09.629719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629649    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629766    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\") pod \"61d1de9d-18ae-45ff-bd47-c3def11ef235\" (UID: \"61d1de9d-18ae-45ff-bd47-c3def11ef235\") "
Oct 30 21:29:09.629893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629822    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.630001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.629980    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.630167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630146    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.630682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630655    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d1de9d-18ae-45ff-bd47-c3def11ef235-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.630790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630684    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.630790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630700    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.630790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630714    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.630790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.630728    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.631555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.631535    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:29:09.631765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.631747    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.631963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.631943    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:09.632028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.631974    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj" pod "61d1de9d-18ae-45ff-bd47-c3def11ef235" (UID: "61d1de9d-18ae-45ff-bd47-c3def11ef235"). InnerVolumeSpecName "kube-api-access-dbfqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:09.732501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.732454    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.732501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.732503    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.732700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.732516    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbfqj\" (UniqueName: \"kubernetes.io/projected/61d1de9d-18ae-45ff-bd47-c3def11ef235-kube-api-access-dbfqj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.732700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:09.732526    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d1de9d-18ae-45ff-bd47-c3def11ef235-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.238085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.238053    7037 generic.go:356] "Generic (PLEG): container finished" podID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerID="dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a" exitCode=0
Oct 30 21:29:10.238257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.238093    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"
Oct 30 21:29:10.238257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.238116    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" event={"ID":"61d1de9d-18ae-45ff-bd47-c3def11ef235","Type":"ContainerDied","Data":"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"}
Oct 30 21:29:10.238257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.238159    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn" event={"ID":"61d1de9d-18ae-45ff-bd47-c3def11ef235","Type":"ContainerDied","Data":"8ceff4daf793551592086363ce30dac40963f69c85ec91ad67dc5505b664979d"}
Oct 30 21:29:10.238257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.238179    7037 scope.go:122] "RemoveContainer" containerID="dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"
Oct 30 21:29:10.249855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.249832    7037 scope.go:122] "RemoveContainer" containerID="dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"
Oct 30 21:29:10.250113 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:29:10.250092    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a\": container with ID starting with dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a not found: ID does not exist" containerID="dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"
Oct 30 21:29:10.250151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.250124    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a"} err="failed to get container status \"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a\": rpc error: code = NotFound desc = could not find container \"dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a\": container with ID starting with dea8c6e847f2dc325b38d09b783fa7695385069b4d38f8c483a4a8c8531b8a3a not found: ID does not exist"
Oct 30 21:29:10.330963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.330924    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:29:10.350586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.350553    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-lm4vn"]
Oct 30 21:29:10.709477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:10.709437    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" path="/var/lib/kubelet/pods/61d1de9d-18ae-45ff-bd47-c3def11ef235/volumes"
Oct 30 21:29:33.672873 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:29:33.672836    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:29:40.983353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:40.983275    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:29:40.984631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:40.984610    7037 cpu_manager.go:414] "containerMap: removing container" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy"
Oct 30 21:29:40.984703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:40.984633    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy"
Oct 30 21:29:40.984845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:40.984833    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d1de9d-18ae-45ff-bd47-c3def11ef235" containerName="istio-proxy"
Oct 30 21:29:40.989244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:40.989220    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.006554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.006525    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-abf04b-serving-cert\""
Oct 30 21:29:41.011665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.011636    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:29:41.047089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.047057    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.047249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.047097    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dh5s\" (UniqueName: \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.047352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.047330    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.047410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.047360    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.148687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.148651    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.148876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.148692    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dh5s\" (UniqueName: \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.148951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.148892    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.148951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.148918    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.149670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.149643    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.151269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.151249    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.151354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.151287    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.183093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.183066    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dh5s\" (UniqueName: \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\") pod \"maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.312455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.311158    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:41.641357 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:29:41.641330    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0c4efa_19b2_44a5_8891_6a55dc1bfb04.slice/crio-14b048ea0f2ecd1177c8d8d92ee277e4593737dfa4a800ca7128b6e566dd6456 WatchSource:0}: Error finding container 14b048ea0f2ecd1177c8d8d92ee277e4593737dfa4a800ca7128b6e566dd6456: Status 404 returned error can't find the container with id 14b048ea0f2ecd1177c8d8d92ee277e4593737dfa4a800ca7128b6e566dd6456
Oct 30 21:29:41.642960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:41.642936    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:29:42.511241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:42.511155    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" event={"ID":"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04","Type":"ContainerStarted","Data":"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"}
Oct 30 21:29:42.511241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:42.511213    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" event={"ID":"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04","Type":"ContainerStarted","Data":"14b048ea0f2ecd1177c8d8d92ee277e4593737dfa4a800ca7128b6e566dd6456"}
Oct 30 21:29:42.511693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:42.511254    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:42.565947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:42.565895    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" podStartSLOduration=2.188783907 podStartE2EDuration="2.565880655s" totalImagesPullingTime="377.096748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:42.560545324 +0000 UTC m=+1725.972084757" watchObservedRunningTime="2026-10-30 21:29:42.565880655 +0000 UTC m=+1725.977420092"
Oct 30 21:29:46.016438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.016402    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:29:46.021857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.021834    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.047909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.047877    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:29:46.052181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.052157    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-67f8e0-serving-cert\""
Oct 30 21:29:46.133317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.133286    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.133516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.133434    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.133594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.133515    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.133660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.133598    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvrl\" (UniqueName: \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.235941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.235902    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.236134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.236006    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.236134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.236064    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.236284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.236255    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvrl\" (UniqueName: \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.236724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.236700    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.238788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.238764    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.239029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.238997    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.258829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.258795    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvrl\" (UniqueName: \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\") pod \"maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.343123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.343080    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:46.665037 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:29:46.665001    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b7b05f_29eb_4184_b612_7fef28999342.slice/crio-57f55826ccccb02c7689152650294e1b17e6e035e0e0023940144383ab9cfc5f WatchSource:0}: Error finding container 57f55826ccccb02c7689152650294e1b17e6e035e0e0023940144383ab9cfc5f: Status 404 returned error can't find the container with id 57f55826ccccb02c7689152650294e1b17e6e035e0e0023940144383ab9cfc5f
Oct 30 21:29:46.666257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:46.666229    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:29:47.555846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:47.555807    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" event={"ID":"61b7b05f-29eb-4184-b612-7fef28999342","Type":"ContainerStarted","Data":"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"}
Oct 30 21:29:47.556341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:47.555889    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" event={"ID":"61b7b05f-29eb-4184-b612-7fef28999342","Type":"ContainerStarted","Data":"57f55826ccccb02c7689152650294e1b17e6e035e0e0023940144383ab9cfc5f"}
Oct 30 21:29:47.556341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:47.555963    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:29:47.608994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:47.608924    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" podStartSLOduration=2.297667283 podStartE2EDuration="2.608910194s" totalImagesPullingTime="311.242911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:47.605386095 +0000 UTC m=+1731.016925541" watchObservedRunningTime="2026-10-30 21:29:47.608910194 +0000 UTC m=+1731.020449630"
Oct 30 21:29:48.525772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:48.525742    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:29:53.569346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:29:53.569315    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:30:00.411458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.411423    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:00.419221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.419191    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:00.419368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.419319    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.430165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.430142    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.484598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.484559    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:30:00.487909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.487871    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:00.492160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.492132    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:00.498098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.498079    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:00.498199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.498167    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.516040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.516008    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:00.522148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.522124    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.525464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.525441    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.525706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.525683    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.525907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.525889    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qppxp\" (UniqueName: \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.526121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.526094    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6s6\" (UniqueName: \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.565575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.565537    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:00.568012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.567988    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:00.571682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.571658    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:00.577583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.577559    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.620986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.620943    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:00.625110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.625047    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627581    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492nb\" (UniqueName: \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627731    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627767    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627794    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627849    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627900    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qppxp\" (UniqueName: \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627954    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.627996    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6s6\" (UniqueName: \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.629010    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.629532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.629181    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.632029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.632007    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:00.632166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.632131    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.637173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.637156    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.669539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.669250    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qppxp\" (UniqueName: \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.672144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.672119    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6s6\" (UniqueName: \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\") pod \"maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.690012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.689979    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:00.698156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.698127    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:00.730001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.729971    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.730188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730097    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492nb\" (UniqueName: \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.730188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730134    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.730188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730170    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.730361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730199    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jp64\" (UniqueName: \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.730361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730308    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.730361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730333    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq64h\" (UniqueName: \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.730361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730356    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kcc\" (UniqueName: \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.730620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730380    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.730620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730427    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.730885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.730863    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.731092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.731073    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.743177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.743149    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:00.750173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.750152    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:00.754858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.754822    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.755689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.755668    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-492nb\" (UniqueName: \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.822978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.822759    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.834396    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835422    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jp64\" (UniqueName: \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835526    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq64h\" (UniqueName: \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kcc\" (UniqueName: \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835591    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835370    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.835860    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.836350    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.836950    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.868843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.839284    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:00.877753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.877289    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq64h\" (UniqueName: \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.877753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.877689    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jp64\" (UniqueName: \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\") pod \"maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:00.879911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.879863    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kcc\" (UniqueName: \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\") pod \"maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.901440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.900963    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:00.954084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.953858    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:00.964647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:00.964611    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:01.110891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.110562    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:01.129230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.129186    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:01.193163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.193131    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:01.200169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.200140    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:01.272432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.272405    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:01.328833 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:01.328798    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c38db6_9750_4843_add2_c44fad28b229.slice/crio-71fcac1df95327324f313c0f25cc416df848199d312889e8f93960141664cd1d WatchSource:0}: Error finding container 71fcac1df95327324f313c0f25cc416df848199d312889e8f93960141664cd1d: Status 404 returned error can't find the container with id 71fcac1df95327324f313c0f25cc416df848199d312889e8f93960141664cd1d
Oct 30 21:30:01.357673 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:01.357616    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0483d920_639f_46ae_9d0f_112afb50c5eb.slice/crio-6e6194e315c2b6b5fe2c05541f47417f99d5a78b348da4031155aef775845650 WatchSource:0}: Error finding container 6e6194e315c2b6b5fe2c05541f47417f99d5a78b348da4031155aef775845650: Status 404 returned error can't find the container with id 6e6194e315c2b6b5fe2c05541f47417f99d5a78b348da4031155aef775845650
Oct 30 21:30:01.357789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.357684    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:01.372531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.372500    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:01.372894 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:01.372870    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e268046_57b3_456e_aeff_f7ec19362294.slice/crio-70b18abbe9ebefb77300318af8c6ad6a043b3bddf55d3a1b0f76707928ce1e09 WatchSource:0}: Error finding container 70b18abbe9ebefb77300318af8c6ad6a043b3bddf55d3a1b0f76707928ce1e09: Status 404 returned error can't find the container with id 70b18abbe9ebefb77300318af8c6ad6a043b3bddf55d3a1b0f76707928ce1e09
Oct 30 21:30:01.672772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.672740    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerStarted","Data":"82e47558c19e3fb926d4fc946e1896d30abc73f5d3ad0abecc1c0cddee98016c"}
Oct 30 21:30:01.673172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.672802    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerStarted","Data":"700d9507e16683926b2d2bef126db920dde7b19209d8c7b5e035f4ec37de2c23"}
Oct 30 21:30:01.674379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.674352    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerStarted","Data":"85b9590b4422cb19ec26af1fdd0dd565acc927725d119f6d7836550473dc90ae"}
Oct 30 21:30:01.675819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.675790    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerStarted","Data":"aac04667e5f8f91bbd35471c2aa1f5d35ded15e2441df42fe5e03b0e58a9d2b9"}
Oct 30 21:30:01.677770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.677582    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerStarted","Data":"14c8a8b8c1b8c9a8081612295e1f63225f9a17f64d3bd4b50e085ec80509e4ea"}
Oct 30 21:30:01.677770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.677678    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerStarted","Data":"4a1b89dcc044ccd1b0abe480f91997caba85061832a152f75ed4e7a4c918877a"}
Oct 30 21:30:01.679230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.679202    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerStarted","Data":"71fcac1df95327324f313c0f25cc416df848199d312889e8f93960141664cd1d"}
Oct 30 21:30:01.680529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.680509    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerStarted","Data":"6e6194e315c2b6b5fe2c05541f47417f99d5a78b348da4031155aef775845650"}
Oct 30 21:30:01.681702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.681682    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerStarted","Data":"70b18abbe9ebefb77300318af8c6ad6a043b3bddf55d3a1b0f76707928ce1e09"}
Oct 30 21:30:01.716736 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.716692    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" podStartSLOduration=1.71667647 podStartE2EDuration="1.71667647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.714325596 +0000 UTC m=+1745.125865032" watchObservedRunningTime="2026-10-30 21:30:01.71667647 +0000 UTC m=+1745.128215939"
Oct 30 21:30:01.760711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:01.760665    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" podStartSLOduration=1.760651032 podStartE2EDuration="1.760651032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.756716864 +0000 UTC m=+1745.168256311" watchObservedRunningTime="2026-10-30 21:30:01.760651032 +0000 UTC m=+1745.172190503"
Oct 30 21:30:02.426802 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:02.426756    7037 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae54cbf_9e80_4ba6_912a_8e548aeca199.slice/crio-b79109d2bf51afcccb1e82249f5f9b95ecd5878274236d4d6875f8b9ec5a0a53.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:30:02.692040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.692001    7037 generic.go:356] "Generic (PLEG): container finished" podID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerID="b79109d2bf51afcccb1e82249f5f9b95ecd5878274236d4d6875f8b9ec5a0a53" exitCode=60
Oct 30 21:30:02.692535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.692094    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerDied","Data":"b79109d2bf51afcccb1e82249f5f9b95ecd5878274236d4d6875f8b9ec5a0a53"}
Oct 30 21:30:02.692535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.692381    7037 scope.go:122] "RemoveContainer" containerID="b79109d2bf51afcccb1e82249f5f9b95ecd5878274236d4d6875f8b9ec5a0a53"
Oct 30 21:30:02.694174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.694142    7037 generic.go:356] "Generic (PLEG): container finished" podID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerID="d420bf0beef3aaad0ae4650abfbb529d5b95150b838240ad81c5ecdcfa0980d0" exitCode=60
Oct 30 21:30:02.694280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.694252    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerDied","Data":"d420bf0beef3aaad0ae4650abfbb529d5b95150b838240ad81c5ecdcfa0980d0"}
Oct 30 21:30:02.694739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.694718    7037 scope.go:122] "RemoveContainer" containerID="d420bf0beef3aaad0ae4650abfbb529d5b95150b838240ad81c5ecdcfa0980d0"
Oct 30 21:30:02.697261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.697243    7037 generic.go:356] "Generic (PLEG): container finished" podID="5c018ac5-609d-42f9-a671-777c071228f7" containerID="14c8a8b8c1b8c9a8081612295e1f63225f9a17f64d3bd4b50e085ec80509e4ea" exitCode=60
Oct 30 21:30:02.697352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.697315    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerDied","Data":"14c8a8b8c1b8c9a8081612295e1f63225f9a17f64d3bd4b50e085ec80509e4ea"}
Oct 30 21:30:02.697681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.697609    7037 scope.go:122] "RemoveContainer" containerID="14c8a8b8c1b8c9a8081612295e1f63225f9a17f64d3bd4b50e085ec80509e4ea"
Oct 30 21:30:02.699241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.699223    7037 generic.go:356] "Generic (PLEG): container finished" podID="62c38db6-9750-4843-add2-c44fad28b229" containerID="6094e7e2f364a0f1c6dd52afa89ddf08a94b6ff05e25a8752dc0affb6d2eb39f" exitCode=60
Oct 30 21:30:02.699308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.699250    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerDied","Data":"6094e7e2f364a0f1c6dd52afa89ddf08a94b6ff05e25a8752dc0affb6d2eb39f"}
Oct 30 21:30:02.699673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.699653    7037 scope.go:122] "RemoveContainer" containerID="6094e7e2f364a0f1c6dd52afa89ddf08a94b6ff05e25a8752dc0affb6d2eb39f"
Oct 30 21:30:02.701319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.701296    7037 generic.go:356] "Generic (PLEG): container finished" podID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerID="16a6846277a74d2961ab4b95aca87f071a9d4d3fbbe677fe8bb647edd252a842" exitCode=60
Oct 30 21:30:02.703857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.703834    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e268046-57b3-456e-aeff-f7ec19362294" containerID="9c40ddb374bde48a960d92b976abf7cb35b45fd9bbe2969771919fbc68504ddf" exitCode=60
Oct 30 21:30:02.713812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.713789    7037 scope.go:122] "RemoveContainer" containerID="16a6846277a74d2961ab4b95aca87f071a9d4d3fbbe677fe8bb647edd252a842"
Oct 30 21:30:02.714378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.714362    7037 scope.go:122] "RemoveContainer" containerID="9c40ddb374bde48a960d92b976abf7cb35b45fd9bbe2969771919fbc68504ddf"
Oct 30 21:30:02.732910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.732876    7037 generic.go:356] "Generic (PLEG): container finished" podID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerID="82e47558c19e3fb926d4fc946e1896d30abc73f5d3ad0abecc1c0cddee98016c" exitCode=60
Oct 30 21:30:02.733434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.733410    7037 scope.go:122] "RemoveContainer" containerID="82e47558c19e3fb926d4fc946e1896d30abc73f5d3ad0abecc1c0cddee98016c"
Oct 30 21:30:02.756757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.756726    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerDied","Data":"16a6846277a74d2961ab4b95aca87f071a9d4d3fbbe677fe8bb647edd252a842"}
Oct 30 21:30:02.756900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.756786    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerDied","Data":"9c40ddb374bde48a960d92b976abf7cb35b45fd9bbe2969771919fbc68504ddf"}
Oct 30 21:30:02.756900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:02.756812    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerDied","Data":"82e47558c19e3fb926d4fc946e1896d30abc73f5d3ad0abecc1c0cddee98016c"}
Oct 30 21:30:03.249442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.249406    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:30:03.257057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.257031    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.309863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.309827    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:30:03.353030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.352996    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c9477c-openshift-default-dockercfg-8hhn7\""
Oct 30 21:30:03.385483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385453    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.385660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385485    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.385825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385801    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.385868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385834    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.385868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385854    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.386018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.385997    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.386093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.386078    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.386134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.386110    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.386171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.386160    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhx4\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487668    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487702    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487724    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487764    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487837    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487860    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.487914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487901    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhx4\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.487980    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488009    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488192    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488237    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488288    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488471    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.488734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.488707    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.490043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.490018    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.490296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.490278    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.515680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.515657    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.528097 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.527855    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhx4\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.577260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.577229    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:03.687891 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:03.687791    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:30:03.751768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.751731    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerStarted","Data":"17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"}
Oct 30 21:30:03.755382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.755347    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerStarted","Data":"c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"}
Oct 30 21:30:03.759036 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.759007    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerStarted","Data":"db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"}
Oct 30 21:30:03.765980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.764648    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerStarted","Data":"d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"}
Oct 30 21:30:03.768512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.767229    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerStarted","Data":"40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"}
Oct 30 21:30:03.775233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.773230    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerStarted","Data":"5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"}
Oct 30 21:30:03.776328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.775552    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerStarted","Data":"9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"}
Oct 30 21:30:03.800039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.799982    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" podStartSLOduration=3.799963833 podStartE2EDuration="3.799963833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.794928682 +0000 UTC m=+1747.206468119" watchObservedRunningTime="2026-10-30 21:30:03.799963833 +0000 UTC m=+1747.211503269"
Oct 30 21:30:03.843101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.843049    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" podStartSLOduration=3.843035367 podStartE2EDuration="3.843035367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.839177668 +0000 UTC m=+1747.250717106" watchObservedRunningTime="2026-10-30 21:30:03.843035367 +0000 UTC m=+1747.254574803"
Oct 30 21:30:03.923158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.923099    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" podStartSLOduration=3.9230796789999998 podStartE2EDuration="3.923079679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.922019513 +0000 UTC m=+1747.333558952" watchObservedRunningTime="2026-10-30 21:30:03.923079679 +0000 UTC m=+1747.334619119"
Oct 30 21:30:03.936794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.936753    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:30:03.938465 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:03.937398    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b37376_68ce_49ba_8d8b_5d050b4cee2e.slice/crio-1f70576faaf26126e04a72760498cb86f599488d39cdb227a9d784f047128719 WatchSource:0}: Error finding container 1f70576faaf26126e04a72760498cb86f599488d39cdb227a9d784f047128719: Status 404 returned error can't find the container with id 1f70576faaf26126e04a72760498cb86f599488d39cdb227a9d784f047128719
Oct 30 21:30:03.941932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.941893    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:03.942049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.941960    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:03.942049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.941994    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:03.968062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:03.967977    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" podStartSLOduration=3.967962834 podStartE2EDuration="3.967962834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.965414672 +0000 UTC m=+1747.376954110" watchObservedRunningTime="2026-10-30 21:30:03.967962834 +0000 UTC m=+1747.379502309"
Oct 30 21:30:04.087801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.081335    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" podStartSLOduration=4.081316112 podStartE2EDuration="4.081316112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.02159272 +0000 UTC m=+1747.433132147" watchObservedRunningTime="2026-10-30 21:30:04.081316112 +0000 UTC m=+1747.492855549"
Oct 30 21:30:04.786303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.786252    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" event={"ID":"c3b37376-68ce-49ba-8d8b-5d050b4cee2e","Type":"ContainerStarted","Data":"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"}
Oct 30 21:30:04.786797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.786339    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" event={"ID":"c3b37376-68ce-49ba-8d8b-5d050b4cee2e","Type":"ContainerStarted","Data":"1f70576faaf26126e04a72760498cb86f599488d39cdb227a9d784f047128719"}
Oct 30 21:30:04.788406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.788370    7037 generic.go:356] "Generic (PLEG): container finished" podID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerID="17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790" exitCode=60
Oct 30 21:30:04.788573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.788456    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerDied","Data":"17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"}
Oct 30 21:30:04.788644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.788564    7037 scope.go:122] "RemoveContainer" containerID="b79109d2bf51afcccb1e82249f5f9b95ecd5878274236d4d6875f8b9ec5a0a53"
Oct 30 21:30:04.788817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.788795    7037 scope.go:122] "RemoveContainer" containerID="17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"
Oct 30 21:30:04.789223 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.789200    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh_odh-ai-gateway-infra(1ae54cbf-9e80-4ba6-912a-8e548aeca199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199"
Oct 30 21:30:04.791041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.791022    7037 generic.go:356] "Generic (PLEG): container finished" podID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerID="c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd" exitCode=60
Oct 30 21:30:04.791143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.791118    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerDied","Data":"c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"}
Oct 30 21:30:04.791363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.791347    7037 scope.go:122] "RemoveContainer" containerID="c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"
Oct 30 21:30:04.791740 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.791679    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x_odh-ai-gateway-infra(fd8ea01b-b933-4f2c-b353-fd10025a75d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9"
Oct 30 21:30:04.793193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.793172    7037 generic.go:356] "Generic (PLEG): container finished" podID="5c018ac5-609d-42f9-a671-777c071228f7" containerID="db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d" exitCode=60
Oct 30 21:30:04.793292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.793251    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerDied","Data":"db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"}
Oct 30 21:30:04.793520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.793506    7037 scope.go:122] "RemoveContainer" containerID="db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"
Oct 30 21:30:04.793844 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.793824    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl_odh-ai-gateway-infra(5c018ac5-609d-42f9-a671-777c071228f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" podUID="5c018ac5-609d-42f9-a671-777c071228f7"
Oct 30 21:30:04.795605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.795582    7037 generic.go:356] "Generic (PLEG): container finished" podID="62c38db6-9750-4843-add2-c44fad28b229" containerID="d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b" exitCode=60
Oct 30 21:30:04.795703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.795679    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerDied","Data":"d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"}
Oct 30 21:30:04.795888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.795872    7037 scope.go:122] "RemoveContainer" containerID="d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"
Oct 30 21:30:04.796161 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.796140    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm_odh-ai-gateway-infra(62c38db6-9750-4843-add2-c44fad28b229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" podUID="62c38db6-9750-4843-add2-c44fad28b229"
Oct 30 21:30:04.797653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.797613    7037 generic.go:356] "Generic (PLEG): container finished" podID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerID="40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607" exitCode=60
Oct 30 21:30:04.797747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.797713    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerDied","Data":"40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"}
Oct 30 21:30:04.797915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.797898    7037 scope.go:122] "RemoveContainer" containerID="40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"
Oct 30 21:30:04.798213 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.798189    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d_odh-ai-gateway-infra(0483d920-639f-46ae-9d0f-112afb50c5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb"
Oct 30 21:30:04.799833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.799799    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e268046-57b3-456e-aeff-f7ec19362294" containerID="5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689" exitCode=60
Oct 30 21:30:04.799918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.799849    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerDied","Data":"5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"}
Oct 30 21:30:04.800066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.800050    7037 scope.go:122] "RemoveContainer" containerID="5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"
Oct 30 21:30:04.800337 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.800317    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl_odh-ai-gateway-infra(8e268046-57b3-456e-aeff-f7ec19362294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" podUID="8e268046-57b3-456e-aeff-f7ec19362294"
Oct 30 21:30:04.802079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.802051    7037 generic.go:356] "Generic (PLEG): container finished" podID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerID="9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c" exitCode=60
Oct 30 21:30:04.802179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.802133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerDied","Data":"9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"}
Oct 30 21:30:04.802315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.802297    7037 scope.go:122] "RemoveContainer" containerID="9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"
Oct 30 21:30:04.802645 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:04.802625    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9_odh-ai-gateway-infra(13facc54-bb4c-4a8c-9391-815f5e83aa50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50"
Oct 30 21:30:04.810845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.810385    7037 scope.go:122] "RemoveContainer" containerID="d420bf0beef3aaad0ae4650abfbb529d5b95150b838240ad81c5ecdcfa0980d0"
Oct 30 21:30:04.827946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.827921    7037 scope.go:122] "RemoveContainer" containerID="14c8a8b8c1b8c9a8081612295e1f63225f9a17f64d3bd4b50e085ec80509e4ea"
Oct 30 21:30:04.846004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.845977    7037 scope.go:122] "RemoveContainer" containerID="6094e7e2f364a0f1c6dd52afa89ddf08a94b6ff05e25a8752dc0affb6d2eb39f"
Oct 30 21:30:04.848763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.848712    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podStartSLOduration=1.848695432 podStartE2EDuration="1.848695432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.841378019 +0000 UTC m=+1748.252917455" watchObservedRunningTime="2026-10-30 21:30:04.848695432 +0000 UTC m=+1748.260234868"
Oct 30 21:30:04.863303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.863280    7037 scope.go:122] "RemoveContainer" containerID="16a6846277a74d2961ab4b95aca87f071a9d4d3fbbe677fe8bb647edd252a842"
Oct 30 21:30:04.878269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.878248    7037 scope.go:122] "RemoveContainer" containerID="9c40ddb374bde48a960d92b976abf7cb35b45fd9bbe2969771919fbc68504ddf"
Oct 30 21:30:04.893105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:04.893081    7037 scope.go:122] "RemoveContainer" containerID="82e47558c19e3fb926d4fc946e1896d30abc73f5d3ad0abecc1c0cddee98016c"
Oct 30 21:30:05.578311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.578277    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:05.579824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.579795    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:05.579976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.579852    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:05.822176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.822146    7037 scope.go:122] "RemoveContainer" containerID="17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"
Oct 30 21:30:05.854210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.854119    7037 scope.go:122] "RemoveContainer" containerID="c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"
Oct 30 21:30:05.855641 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.854119    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh_odh-ai-gateway-infra(1ae54cbf-9e80-4ba6-912a-8e548aeca199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199"
Oct 30 21:30:05.856937 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.856906    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x_odh-ai-gateway-infra(fd8ea01b-b933-4f2c-b353-fd10025a75d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9"
Oct 30 21:30:05.861046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.861012    7037 scope.go:122] "RemoveContainer" containerID="db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"
Oct 30 21:30:05.861612 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.861581    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl_odh-ai-gateway-infra(5c018ac5-609d-42f9-a671-777c071228f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" podUID="5c018ac5-609d-42f9-a671-777c071228f7"
Oct 30 21:30:05.863138 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.863121    7037 scope.go:122] "RemoveContainer" containerID="d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"
Oct 30 21:30:05.863426 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.863398    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm_odh-ai-gateway-infra(62c38db6-9750-4843-add2-c44fad28b229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" podUID="62c38db6-9750-4843-add2-c44fad28b229"
Oct 30 21:30:05.864937 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.864920    7037 scope.go:122] "RemoveContainer" containerID="40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"
Oct 30 21:30:05.865176 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.865154    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d_odh-ai-gateway-infra(0483d920-639f-46ae-9d0f-112afb50c5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb"
Oct 30 21:30:05.866765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.866748    7037 scope.go:122] "RemoveContainer" containerID="5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"
Oct 30 21:30:05.867006 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.866985    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl_odh-ai-gateway-infra(8e268046-57b3-456e-aeff-f7ec19362294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" podUID="8e268046-57b3-456e-aeff-f7ec19362294"
Oct 30 21:30:05.869578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:05.869558    7037 scope.go:122] "RemoveContainer" containerID="9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"
Oct 30 21:30:05.869859 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:05.869838    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9_odh-ai-gateway-infra(13facc54-bb4c-4a8c-9391-815f5e83aa50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50"
Oct 30 21:30:06.577834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:06.577801    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:06.577989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:06.577853    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:06.654077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:06.654040    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:30:06.654369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:06.654319    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy" containerID="cri-o://48dd3c538e283dc741e32efd57bf2a22ce5bf61e8a1bf4dd117c61a857436491" gracePeriod=30
Oct 30 21:30:07.578651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:07.578611    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:07.579064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:07.578674    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:08.578513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:08.578459    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:08.578691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:08.578526    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:09.578244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:09.578206    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:09.578419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:09.578264    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:10.578472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:10.578437    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:30:10.578912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:10.578503    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Oct 30 21:30:11.583218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:11.583190    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:11.583708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:11.583688    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:11.584352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:11.584329    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:30:11.924428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:11.924396    7037 generic.go:356] "Generic (PLEG): container finished" podID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerID="48dd3c538e283dc741e32efd57bf2a22ce5bf61e8a1bf4dd117c61a857436491" exitCode=0
Oct 30 21:30:11.925254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:11.925201    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" event={"ID":"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0","Type":"ContainerDied","Data":"48dd3c538e283dc741e32efd57bf2a22ce5bf61e8a1bf4dd117c61a857436491"}
Oct 30 21:30:12.003007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.002979    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:30:12.079642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.079608    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.079642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.079649    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.079951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.079759    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.079951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.079861    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.080076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:12.080135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080099    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:12.080260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080245    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:12.080439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080418    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.080522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080468    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.080591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080559    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.080648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080591    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.080648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.080632    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\") pod \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\" (UID: \"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0\") "
Oct 30 21:30:12.081694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.081668    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.081821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.081696    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.081821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.081710    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.082597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.082535    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:12.082988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.082956    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:12.083483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.083457    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "kube-api-access-j2jcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:12.085330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.085306    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:12.085443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.085420    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:12.087226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.087202    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token" pod "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" (UID: "f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:12.183947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183867    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jcs\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-kube-api-access-j2jcs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.183947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183901    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.183947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183916    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.183947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183929    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.183947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183941    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.184237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.183956    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:12.933908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.933876    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"
Oct 30 21:30:12.934320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.933912    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6" event={"ID":"f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0","Type":"ContainerDied","Data":"8ab02323b5518b13ade98e6de6b1ec594a6331bd760db3bc846fe85a47141a3f"}
Oct 30 21:30:12.934320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.933974    7037 scope.go:122] "RemoveContainer" containerID="48dd3c538e283dc741e32efd57bf2a22ce5bf61e8a1bf4dd117c61a857436491"
Oct 30 21:30:12.988527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.988441    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:30:12.992801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:12.992776    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-vp9n6"]
Oct 30 21:30:14.711684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:14.711653    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" path="/var/lib/kubelet/pods/f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0/volumes"
Oct 30 21:30:15.700244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:15.700206    7037 scope.go:122] "RemoveContainer" containerID="17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"
Oct 30 21:30:15.700414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:15.700381    7037 scope.go:122] "RemoveContainer" containerID="9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"
Oct 30 21:30:15.700526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:15.700509    7037 scope.go:122] "RemoveContainer" containerID="c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"
Oct 30 21:30:15.700620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:15.700602    7037 scope.go:122] "RemoveContainer" containerID="db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"
Oct 30 21:30:15.700783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:15.700748    7037 scope.go:122] "RemoveContainer" containerID="d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"
Oct 30 21:30:16.972877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.972851    7037 generic.go:356] "Generic (PLEG): container finished" podID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerID="98210099c86aa1c09afbee4b42de1213770b2538a080de2ed7e7b601b47815e1" exitCode=60
Oct 30 21:30:16.973227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.972930    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerDied","Data":"98210099c86aa1c09afbee4b42de1213770b2538a080de2ed7e7b601b47815e1"}
Oct 30 21:30:16.973227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.972996    7037 scope.go:122] "RemoveContainer" containerID="17b9ff8375a8650a92e05c84fb36d1cf86275fff8d9fed3166a019717e6e1790"
Oct 30 21:30:16.973410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.973395    7037 scope.go:122] "RemoveContainer" containerID="98210099c86aa1c09afbee4b42de1213770b2538a080de2ed7e7b601b47815e1"
Oct 30 21:30:16.973851 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:16.973822    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh_odh-ai-gateway-infra(1ae54cbf-9e80-4ba6-912a-8e548aeca199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199"
Oct 30 21:30:16.975285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.975254    7037 generic.go:356] "Generic (PLEG): container finished" podID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerID="c131cd28d002bc3edf97823ba6665fd64facd034fd76b125e886c808afec3b9f" exitCode=60
Oct 30 21:30:16.975390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.975309    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerDied","Data":"c131cd28d002bc3edf97823ba6665fd64facd034fd76b125e886c808afec3b9f"}
Oct 30 21:30:16.975712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.975694    7037 scope.go:122] "RemoveContainer" containerID="c131cd28d002bc3edf97823ba6665fd64facd034fd76b125e886c808afec3b9f"
Oct 30 21:30:16.976082 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:16.976056    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x_odh-ai-gateway-infra(fd8ea01b-b933-4f2c-b353-fd10025a75d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9"
Oct 30 21:30:16.977310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.977288    7037 generic.go:356] "Generic (PLEG): container finished" podID="5c018ac5-609d-42f9-a671-777c071228f7" containerID="66cf2a13b73b9f7af147f8017772f9920c9884eb21cc7fdf990cbd36e2455064" exitCode=60
Oct 30 21:30:16.977412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.977363    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerDied","Data":"66cf2a13b73b9f7af147f8017772f9920c9884eb21cc7fdf990cbd36e2455064"}
Oct 30 21:30:16.977758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.977742    7037 scope.go:122] "RemoveContainer" containerID="66cf2a13b73b9f7af147f8017772f9920c9884eb21cc7fdf990cbd36e2455064"
Oct 30 21:30:16.978103 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:16.978080    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl_odh-ai-gateway-infra(5c018ac5-609d-42f9-a671-777c071228f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" podUID="5c018ac5-609d-42f9-a671-777c071228f7"
Oct 30 21:30:16.979466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.979448    7037 generic.go:356] "Generic (PLEG): container finished" podID="62c38db6-9750-4843-add2-c44fad28b229" containerID="78c30ca80f97a8d0cd36479c506a154499b205813f9b0cfd0f1c23fce6c62eef" exitCode=60
Oct 30 21:30:16.979572 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.979514    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerDied","Data":"78c30ca80f97a8d0cd36479c506a154499b205813f9b0cfd0f1c23fce6c62eef"}
Oct 30 21:30:16.979821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.979800    7037 scope.go:122] "RemoveContainer" containerID="78c30ca80f97a8d0cd36479c506a154499b205813f9b0cfd0f1c23fce6c62eef"
Oct 30 21:30:16.980095 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:16.980079    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm_odh-ai-gateway-infra(62c38db6-9750-4843-add2-c44fad28b229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" podUID="62c38db6-9750-4843-add2-c44fad28b229"
Oct 30 21:30:16.981502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.981471    7037 generic.go:356] "Generic (PLEG): container finished" podID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerID="9bcfa61eede6b15c621d4d8bc260783e63de71cf1d609d4e6236f50057802144" exitCode=60
Oct 30 21:30:16.981581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.981518    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerDied","Data":"9bcfa61eede6b15c621d4d8bc260783e63de71cf1d609d4e6236f50057802144"}
Oct 30 21:30:16.981831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:16.981817    7037 scope.go:122] "RemoveContainer" containerID="9bcfa61eede6b15c621d4d8bc260783e63de71cf1d609d4e6236f50057802144"
Oct 30 21:30:16.982056 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:16.982040    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9_odh-ai-gateway-infra(13facc54-bb4c-4a8c-9391-815f5e83aa50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50"
Oct 30 21:30:17.113140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.113118    7037 scope.go:122] "RemoveContainer" containerID="c5197e0bd586a38bf106fbbf821fbf8738737af6e8b7e2a1bed30de5ea657dfd"
Oct 30 21:30:17.127953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.127929    7037 scope.go:122] "RemoveContainer" containerID="db49f9b2898a97543c125794ba98a4827f5d73e04607558276f719a73fbaf31d"
Oct 30 21:30:17.141266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.141249    7037 scope.go:122] "RemoveContainer" containerID="d4258237bf269bd20d8af1a04f042b00c63c57fa0d2433d3f77fb4ba9f8c926b"
Oct 30 21:30:17.155416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.155397    7037 scope.go:122] "RemoveContainer" containerID="9a6e942c66ecbcf723f1a5a810e7c91778525f4de2f8307f29f5b10c8fc94d3c"
Oct 30 21:30:17.699856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.699830    7037 scope.go:122] "RemoveContainer" containerID="40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"
Oct 30 21:30:17.700044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:17.699982    7037 scope.go:122] "RemoveContainer" containerID="5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"
Oct 30 21:30:18.092513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.087749    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:18.111179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.103695    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:18.123560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.122270    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:18.162061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.161821    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:18.211835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.206238    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:18.512853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.512817    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:18.619872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.619678    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\") pod \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") "
Oct 30 21:30:18.620053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.619970    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\") pod \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\" (UID: \"1ae54cbf-9e80-4ba6-912a-8e548aeca199\") "
Oct 30 21:30:18.620195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.620162    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca" pod "1ae54cbf-9e80-4ba6-912a-8e548aeca199" (UID: "1ae54cbf-9e80-4ba6-912a-8e548aeca199"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.620762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.620736    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae54cbf-9e80-4ba6-912a-8e548aeca199-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.622560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.622532    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b" pod "1ae54cbf-9e80-4ba6-912a-8e548aeca199" (UID: "1ae54cbf-9e80-4ba6-912a-8e548aeca199"). InnerVolumeSpecName "kube-api-access-tjv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.652217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.651854    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:18.722783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.722575    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/1ae54cbf-9e80-4ba6-912a-8e548aeca199-kube-api-access-tjv2b\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.751531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.736537    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:18.792826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.792736    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:18.799375 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.799349    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.830224    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\" (UniqueName: \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\") pod \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") "
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.841599    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\" (UniqueName: \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\") pod \"5c018ac5-609d-42f9-a671-777c071228f7\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") "
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.841750    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\") pod \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\" (UID: \"fd8ea01b-b933-4f2c-b353-fd10025a75d9\") "
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.841846    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\") pod \"5c018ac5-609d-42f9-a671-777c071228f7\" (UID: \"5c018ac5-609d-42f9-a671-777c071228f7\") "
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.844808    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca" pod "fd8ea01b-b933-4f2c-b353-fd10025a75d9" (UID: "fd8ea01b-b933-4f2c-b353-fd10025a75d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.845139    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca" pod "5c018ac5-609d-42f9-a671-777c071228f7" (UID: "5c018ac5-609d-42f9-a671-777c071228f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.851612    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:30:18.852601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.852464    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" containerID="cri-o://3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655" gracePeriod=30
Oct 30 21:30:18.870031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.869978    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp" pod "5c018ac5-609d-42f9-a671-777c071228f7" (UID: "5c018ac5-609d-42f9-a671-777c071228f7"). InnerVolumeSpecName "kube-api-access-qppxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.871629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.871045    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb" pod "fd8ea01b-b933-4f2c-b353-fd10025a75d9" (UID: "fd8ea01b-b933-4f2c-b353-fd10025a75d9"). InnerVolumeSpecName "kube-api-access-492nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.959027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.955428    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\" (UniqueName: \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\") pod \"13facc54-bb4c-4a8c-9391-815f5e83aa50\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") "
Oct 30 21:30:18.959027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.955986    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\") pod \"13facc54-bb4c-4a8c-9391-815f5e83aa50\" (UID: \"13facc54-bb4c-4a8c-9391-815f5e83aa50\") "
Oct 30 21:30:18.959027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.956050    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\") pod \"62c38db6-9750-4843-add2-c44fad28b229\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") "
Oct 30 21:30:18.959027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.956145    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\" (UniqueName: \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\") pod \"62c38db6-9750-4843-add2-c44fad28b229\" (UID: \"62c38db6-9750-4843-add2-c44fad28b229\") "
Oct 30 21:30:18.960894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.960859    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca" pod "13facc54-bb4c-4a8c-9391-815f5e83aa50" (UID: "13facc54-bb4c-4a8c-9391-815f5e83aa50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.962447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.962419    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca" pod "62c38db6-9750-4843-add2-c44fad28b229" (UID: "62c38db6-9750-4843-add2-c44fad28b229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.988578    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13facc54-bb4c-4a8c-9391-815f5e83aa50-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.989395    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c38db6-9750-4843-add2-c44fad28b229-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.989430    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ea01b-b933-4f2c-b353-fd10025a75d9-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.989456    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c018ac5-609d-42f9-a671-777c071228f7-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.989474    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492nb\" (UniqueName: \"kubernetes.io/projected/fd8ea01b-b933-4f2c-b353-fd10025a75d9-kube-api-access-492nb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.991533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.989503    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qppxp\" (UniqueName: \"kubernetes.io/projected/5c018ac5-609d-42f9-a671-777c071228f7-kube-api-access-qppxp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.999523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.996631    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6" pod "13facc54-bb4c-4a8c-9391-815f5e83aa50" (UID: "13facc54-bb4c-4a8c-9391-815f5e83aa50"). InnerVolumeSpecName "kube-api-access-jd6s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.999523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:18.998703    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc" pod "62c38db6-9750-4843-add2-c44fad28b229" (UID: "62c38db6-9750-4843-add2-c44fad28b229"). InnerVolumeSpecName "kube-api-access-n4kcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:19.064684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.064647    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:30:19.065245 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.065209    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy" containerID="cri-o://d343a82a8d9d19ea8ae939e77b2ca8d722ad573d647401a0af418002a6a3f96f" gracePeriod=30
Oct 30 21:30:19.091452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.091417    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kcc\" (UniqueName: \"kubernetes.io/projected/62c38db6-9750-4843-add2-c44fad28b229-kube-api-access-n4kcc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:19.096187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.095738    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6s6\" (UniqueName: \"kubernetes.io/projected/13facc54-bb4c-4a8c-9391-815f5e83aa50-kube-api-access-jd6s6\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:19.103886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.103854    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e268046-57b3-456e-aeff-f7ec19362294" containerID="b341d068671f3dcf3a7a913f05fb9e72040c4ae7d76302764138edfffd4f6558" exitCode=60
Oct 30 21:30:19.104344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.104196    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerDied","Data":"b341d068671f3dcf3a7a913f05fb9e72040c4ae7d76302764138edfffd4f6558"}
Oct 30 21:30:19.109214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.106915    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"
Oct 30 21:30:19.111907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.109098    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"
Oct 30 21:30:19.127439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.127407    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9" event={"ID":"13facc54-bb4c-4a8c-9391-815f5e83aa50","Type":"ContainerDied","Data":"700d9507e16683926b2d2bef126db920dde7b19209d8c7b5e035f4ec37de2c23"}
Oct 30 21:30:19.129193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.127668    7037 scope.go:122] "RemoveContainer" containerID="5e2c2b2ca2cdab3b1bab9b27e31838f13e820f76e2cd33e96f2e41b3e4af7689"
Oct 30 21:30:19.129429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.129414    7037 scope.go:122] "RemoveContainer" containerID="b341d068671f3dcf3a7a913f05fb9e72040c4ae7d76302764138edfffd4f6558"
Oct 30 21:30:19.130282 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:19.130257    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl_odh-ai-gateway-infra(8e268046-57b3-456e-aeff-f7ec19362294)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" podUID="8e268046-57b3-456e-aeff-f7ec19362294"
Oct 30 21:30:19.135550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.131817    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"
Oct 30 21:30:19.150323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.145680    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh" event={"ID":"1ae54cbf-9e80-4ba6-912a-8e548aeca199","Type":"ContainerDied","Data":"85b9590b4422cb19ec26af1fdd0dd565acc927725d119f6d7836550473dc90ae"}
Oct 30 21:30:19.150323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.145720    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x" event={"ID":"fd8ea01b-b933-4f2c-b353-fd10025a75d9","Type":"ContainerDied","Data":"aac04667e5f8f91bbd35471c2aa1f5d35ded15e2441df42fe5e03b0e58a9d2b9"}
Oct 30 21:30:19.157073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.157024    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl" event={"ID":"5c018ac5-609d-42f9-a671-777c071228f7","Type":"ContainerDied","Data":"4a1b89dcc044ccd1b0abe480f91997caba85061832a152f75ed4e7a4c918877a"}
Oct 30 21:30:19.157394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.157374    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"
Oct 30 21:30:19.160340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.160315    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm" event={"ID":"62c38db6-9750-4843-add2-c44fad28b229","Type":"ContainerDied","Data":"71fcac1df95327324f313c0f25cc416df848199d312889e8f93960141664cd1d"}
Oct 30 21:30:19.161757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.160839    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"
Oct 30 21:30:19.166016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.165561    7037 generic.go:356] "Generic (PLEG): container finished" podID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerID="db507e10b523a2b29cd041da302761af52abb020e3b441d58065f69fc44e04b1" exitCode=60
Oct 30 21:30:19.166016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.165614    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerDied","Data":"db507e10b523a2b29cd041da302761af52abb020e3b441d58065f69fc44e04b1"}
Oct 30 21:30:19.166182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.166132    7037 scope.go:122] "RemoveContainer" containerID="db507e10b523a2b29cd041da302761af52abb020e3b441d58065f69fc44e04b1"
Oct 30 21:30:19.167484 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:19.166469    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d_odh-ai-gateway-infra(0483d920-639f-46ae-9d0f-112afb50c5eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb"
Oct 30 21:30:19.179208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.179186    7037 scope.go:122] "RemoveContainer" containerID="9bcfa61eede6b15c621d4d8bc260783e63de71cf1d609d4e6236f50057802144"
Oct 30 21:30:19.195481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.195458    7037 scope.go:122] "RemoveContainer" containerID="98210099c86aa1c09afbee4b42de1213770b2538a080de2ed7e7b601b47815e1"
Oct 30 21:30:19.210704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.210622    7037 scope.go:122] "RemoveContainer" containerID="c131cd28d002bc3edf97823ba6665fd64facd034fd76b125e886c808afec3b9f"
Oct 30 21:30:19.239080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.239059    7037 scope.go:122] "RemoveContainer" containerID="66cf2a13b73b9f7af147f8017772f9920c9884eb21cc7fdf990cbd36e2455064"
Oct 30 21:30:19.250042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.250016    7037 scope.go:122] "RemoveContainer" containerID="78c30ca80f97a8d0cd36479c506a154499b205813f9b0cfd0f1c23fce6c62eef"
Oct 30 21:30:19.264729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.262415    7037 scope.go:122] "RemoveContainer" containerID="40687ba90e7213c438bc6b60e87a8097b23bcef3eef21608c756898ef0f1b607"
Oct 30 21:30:19.298017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.297987    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:19.308307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.308274    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3b28bc-29889930-trvqm"]
Oct 30 21:30:19.382074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.381991    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:19.390048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.390017    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c5145c-29889930-2lbn9"]
Oct 30 21:30:19.431285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.431248    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:19.436027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.435999    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889930-kgv5x"]
Oct 30 21:30:19.513442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.513400    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:19.518904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.518875    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889930-8zdwh"]
Oct 30 21:30:19.575504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.575453    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:19.592194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:19.592154    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889930-nvlcl"]
Oct 30 21:30:20.220884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.220847    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:20.472946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.472878    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:20.488739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.488709    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:20.520405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.520367    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\") pod \"8e268046-57b3-456e-aeff-f7ec19362294\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") "
Oct 30 21:30:20.520785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.520765    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\" (UniqueName: \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\") pod \"8e268046-57b3-456e-aeff-f7ec19362294\" (UID: \"8e268046-57b3-456e-aeff-f7ec19362294\") "
Oct 30 21:30:20.536017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.535011    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca" pod "8e268046-57b3-456e-aeff-f7ec19362294" (UID: "8e268046-57b3-456e-aeff-f7ec19362294"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.543003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.542155    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64" pod "8e268046-57b3-456e-aeff-f7ec19362294" (UID: "8e268046-57b3-456e-aeff-f7ec19362294"). InnerVolumeSpecName "kube-api-access-7jp64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.632365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.632308    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jp64\" (UniqueName: \"kubernetes.io/projected/8e268046-57b3-456e-aeff-f7ec19362294-kube-api-access-7jp64\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.632365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.632367    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268046-57b3-456e-aeff-f7ec19362294-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.719959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.718833    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" path="/var/lib/kubelet/pods/13facc54-bb4c-4a8c-9391-815f5e83aa50/volumes"
Oct 30 21:30:20.719959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.719613    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" path="/var/lib/kubelet/pods/1ae54cbf-9e80-4ba6-912a-8e548aeca199/volumes"
Oct 30 21:30:20.720351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.720328    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c018ac5-609d-42f9-a671-777c071228f7" path="/var/lib/kubelet/pods/5c018ac5-609d-42f9-a671-777c071228f7/volumes"
Oct 30 21:30:20.721358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.721330    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c38db6-9750-4843-add2-c44fad28b229" path="/var/lib/kubelet/pods/62c38db6-9750-4843-add2-c44fad28b229/volumes"
Oct 30 21:30:20.721988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.721964    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" path="/var/lib/kubelet/pods/fd8ea01b-b933-4f2c-b353-fd10025a75d9/volumes"
Oct 30 21:30:20.740113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.740089    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:20.834312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.834279    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\") pod \"0483d920-639f-46ae-9d0f-112afb50c5eb\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") "
Oct 30 21:30:20.834547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.834338    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\" (UniqueName: \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\") pod \"0483d920-639f-46ae-9d0f-112afb50c5eb\" (UID: \"0483d920-639f-46ae-9d0f-112afb50c5eb\") "
Oct 30 21:30:20.834743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.834715    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca" pod "0483d920-639f-46ae-9d0f-112afb50c5eb" (UID: "0483d920-639f-46ae-9d0f-112afb50c5eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.835679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.835528    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0483d920-639f-46ae-9d0f-112afb50c5eb-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.837719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.837685    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h" pod "0483d920-639f-46ae-9d0f-112afb50c5eb" (UID: "0483d920-639f-46ae-9d0f-112afb50c5eb"). InnerVolumeSpecName "kube-api-access-jq64h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.937054 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:20.937010    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq64h\" (UniqueName: \"kubernetes.io/projected/0483d920-639f-46ae-9d0f-112afb50c5eb-kube-api-access-jq64h\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.226880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.226846    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d" event={"ID":"0483d920-639f-46ae-9d0f-112afb50c5eb","Type":"ContainerDied","Data":"6e6194e315c2b6b5fe2c05541f47417f99d5a78b348da4031155aef775845650"}
Oct 30 21:30:21.227961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.227359    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"
Oct 30 21:30:21.228092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.227952    7037 scope.go:122] "RemoveContainer" containerID="db507e10b523a2b29cd041da302761af52abb020e3b441d58065f69fc44e04b1"
Oct 30 21:30:21.231368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.231333    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl" event={"ID":"8e268046-57b3-456e-aeff-f7ec19362294","Type":"ContainerDied","Data":"70b18abbe9ebefb77300318af8c6ad6a043b3bddf55d3a1b0f76707928ce1e09"}
Oct 30 21:30:21.232788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.232740    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"
Oct 30 21:30:21.257906 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.253926    7037 scope.go:122] "RemoveContainer" containerID="b341d068671f3dcf3a7a913f05fb9e72040c4ae7d76302764138edfffd4f6558"
Oct 30 21:30:21.288740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.288713    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:21.294198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.294172    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889930-9lf8d"]
Oct 30 21:30:21.333062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.333023    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:21.339155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.339131    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-200d11-29889930-7qbvl"]
Oct 30 21:30:21.702294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.702251    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:30:21.702483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:21.702459    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy" containerID="cri-o://c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437" gracePeriod=30
Oct 30 21:30:22.713074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:22.713031    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" path="/var/lib/kubelet/pods/0483d920-639f-46ae-9d0f-112afb50c5eb/volumes"
Oct 30 21:30:22.713822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:22.713795    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e268046-57b3-456e-aeff-f7ec19362294" path="/var/lib/kubelet/pods/8e268046-57b3-456e-aeff-f7ec19362294/volumes"
Oct 30 21:30:23.903657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:23.903592    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:30:23.904110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:23.903669    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 21:30:24.242319 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.242291    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:30:24.267164 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.267128    7037 generic.go:356] "Generic (PLEG): container finished" podID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerID="d343a82a8d9d19ea8ae939e77b2ca8d722ad573d647401a0af418002a6a3f96f" exitCode=0
Oct 30 21:30:24.267337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.267202    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" event={"ID":"de1e9aec-c264-4d18-a5af-f930f8fdea14","Type":"ContainerDied","Data":"d343a82a8d9d19ea8ae939e77b2ca8d722ad573d647401a0af418002a6a3f96f"}
Oct 30 21:30:24.269043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.269015    7037 generic.go:356] "Generic (PLEG): container finished" podID="782e5973-dc08-4e8a-916a-b99deef909c0" containerID="3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655" exitCode=0
Oct 30 21:30:24.269195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.269071    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"
Oct 30 21:30:24.269195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.269100    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" event={"ID":"782e5973-dc08-4e8a-916a-b99deef909c0","Type":"ContainerDied","Data":"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"}
Oct 30 21:30:24.269195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.269157    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf" event={"ID":"782e5973-dc08-4e8a-916a-b99deef909c0","Type":"ContainerDied","Data":"a0e0fddd6b35bb7bc7e4ccb47f863e67219359d1491d16de7e4cfd3a62f27dfa"}
Oct 30 21:30:24.269195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.269191    7037 scope.go:122] "RemoveContainer" containerID="3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"
Oct 30 21:30:24.320955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.320924    7037 scope.go:122] "RemoveContainer" containerID="3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"
Oct 30 21:30:24.321963 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:24.321386    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655\": container with ID starting with 3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655 not found: ID does not exist" containerID="3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"
Oct 30 21:30:24.321963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.321423    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655"} err="failed to get container status \"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655\": rpc error: code = NotFound desc = could not find container \"3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655\": container with ID starting with 3a3bab21baa2a5127b35f46ff778ee1fe61aa51c43e36bdd37a45ec6505d4655 not found: ID does not exist"
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.408230    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417407    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417479    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417523    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417674    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417736    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417781    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417881    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417918    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.417975    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\") pod \"782e5973-dc08-4e8a-916a-b99deef909c0\" (UID: \"782e5973-dc08-4e8a-916a-b99deef909c0\") "
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.419537    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.419535    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.419723    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.419765    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.422023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.420134    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.423059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.423038    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "kube-api-access-2rmnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.423224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.423198    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:24.426832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.426802    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.426943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.426877    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token" pod "782e5973-dc08-4e8a-916a-b99deef909c0" (UID: "782e5973-dc08-4e8a-916a-b99deef909c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.520214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520177    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520240    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520282    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520387    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520426    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520586    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520634    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520684    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.520766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520745    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\") pod \"de1e9aec-c264-4d18-a5af-f930f8fdea14\" (UID: \"de1e9aec-c264-4d18-a5af-f930f8fdea14\") "
Oct 30 21:30:24.521152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520566    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.521152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520738    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.521152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.520833    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.521152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521088    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.521152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521124    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.521538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521518    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1e9aec-c264-4d18-a5af-f930f8fdea14-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521546    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521562    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521580    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521603    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/782e5973-dc08-4e8a-916a-b99deef909c0-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521617    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521646    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521657    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521669    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521681    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rmnd\" (UniqueName: \"kubernetes.io/projected/782e5973-dc08-4e8a-916a-b99deef909c0-kube-api-access-2rmnd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521693    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/782e5973-dc08-4e8a-916a-b99deef909c0-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521707    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521719    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/782e5973-dc08-4e8a-916a-b99deef909c0-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.521952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.521733    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.522818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.522795    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.523123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.523097    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "kube-api-access-xq59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.523123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.523108    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:24.523252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.523097    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy" pod "de1e9aec-c264-4d18-a5af-f930f8fdea14" (UID: "de1e9aec-c264-4d18-a5af-f930f8fdea14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:24.624199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.624166    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.624199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.624198    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.624392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.624210    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1e9aec-c264-4d18-a5af-f930f8fdea14-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.624392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.624222    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq59j\" (UniqueName: \"kubernetes.io/projected/de1e9aec-c264-4d18-a5af-f930f8fdea14-kube-api-access-xq59j\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:24.631414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.631377    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:30:24.637612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.637575    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-zhrnf"]
Oct 30 21:30:24.710621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:24.710545    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" path="/var/lib/kubelet/pods/782e5973-dc08-4e8a-916a-b99deef909c0/volumes"
Oct 30 21:30:25.281416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:25.281373    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql" event={"ID":"de1e9aec-c264-4d18-a5af-f930f8fdea14","Type":"ContainerDied","Data":"8c8b1771c42d592bb050bc617c6a7a3df27770af15292143b3dc27601447a2e5"}
Oct 30 21:30:25.281992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:25.281432    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"
Oct 30 21:30:25.281992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:25.281462    7037 scope.go:122] "RemoveContainer" containerID="d343a82a8d9d19ea8ae939e77b2ca8d722ad573d647401a0af418002a6a3f96f"
Oct 30 21:30:25.360980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:25.360921    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:30:25.361417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:25.361390    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-xr6ql"]
Oct 30 21:30:26.712800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:26.712766    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" path="/var/lib/kubelet/pods/de1e9aec-c264-4d18-a5af-f930f8fdea14/volumes"
Oct 30 21:30:26.992713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:26.992681    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:30:27.063236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063201    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063274    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063328    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063381    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063442    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063551    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:27.063610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063570    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063604    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063765    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.063844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063829    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\") pod \"f995af46-5f95-4be6-85c1-4b468404c1b0\" (UID: \"f995af46-5f95-4be6-85c1-4b468404c1b0\") "
Oct 30 21:30:27.064566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.064542    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.064837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.063765    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:27.064837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.064029    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:27.065042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.064192    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:27.065042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.064803    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:27.067105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.067077    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:27.067894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.067869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:27.074193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.074146    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "kube-api-access-hfpfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:27.076595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.076569    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy" pod "f995af46-5f95-4be6-85c1-4b468404c1b0" (UID: "f995af46-5f95-4be6-85c1-4b468404c1b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:27.165808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165774    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.165808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165806    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfpfd\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-kube-api-access-hfpfd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.165808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165816    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.166013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165825    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.166013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165833    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.166013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165843    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f995af46-5f95-4be6-85c1-4b468404c1b0-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.166013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165850    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f995af46-5f95-4be6-85c1-4b468404c1b0-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.166013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.165857    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f995af46-5f95-4be6-85c1-4b468404c1b0-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:30:27.302425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.302332    7037 generic.go:356] "Generic (PLEG): container finished" podID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerID="c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437" exitCode=0
Oct 30 21:30:27.302425 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.302403    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"
Oct 30 21:30:27.302724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.302430    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" event={"ID":"f995af46-5f95-4be6-85c1-4b468404c1b0","Type":"ContainerDied","Data":"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"}
Oct 30 21:30:27.302724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.302538    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm" event={"ID":"f995af46-5f95-4be6-85c1-4b468404c1b0","Type":"ContainerDied","Data":"203d114ff63d4e2febb646db751e34fac969dbd275c38dc6b4b72467736ce0e9"}
Oct 30 21:30:27.302724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.302567    7037 scope.go:122] "RemoveContainer" containerID="c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"
Oct 30 21:30:27.315701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.315676    7037 scope.go:122] "RemoveContainer" containerID="c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"
Oct 30 21:30:27.315966 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:27.315950    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437\": container with ID starting with c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437 not found: ID does not exist" containerID="c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"
Oct 30 21:30:27.316019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.315973    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437"} err="failed to get container status \"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437\": rpc error: code = NotFound desc = could not find container \"c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437\": container with ID starting with c00c5181529cc15710c4670652cbe3a26742d0c7f194155403bc93e50d65c437 not found: ID does not exist"
Oct 30 21:30:27.360444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.360411    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:30:27.371527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:27.371480    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7kdrm"]
Oct 30 21:30:28.712391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:28.712356    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" path="/var/lib/kubelet/pods/f995af46-5f95-4be6-85c1-4b468404c1b0/volumes"
Oct 30 21:30:32.379144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.379109    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:30:32.380178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380153    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380178    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380191    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380198    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380212    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380220    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380231    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380237    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380258    7037 cpu_manager.go:414] "containerMap: removing container" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380264    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380274    7037 cpu_manager.go:414] "containerMap: removing container" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380281    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380294    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380300    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380313    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.380314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380319    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380341    7037 cpu_manager.go:414] "containerMap: removing container" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380348    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380364    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380371    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380383    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380391    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380404    7037 cpu_manager.go:414] "containerMap: removing container" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380411    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380425    7037 cpu_manager.go:414] "containerMap: removing container" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380432    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380449    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380457    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380480    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380503    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380517    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380524    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380539    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380546    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380562    7037 cpu_manager.go:414] "containerMap: removing container" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380571    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380581    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380588    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380600    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380606    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380628    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380635    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380832    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e82098-0ef4-4db4-bf3d-fbf5a3e59bf0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380849    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f995af46-5f95-4be6-85c1-4b468404c1b0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380858    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380867    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380877    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380887    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380896    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380904    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380913    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380922    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380931    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380944    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380952    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380960    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380968    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268046-57b3-456e-aeff-f7ec19362294" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380982    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8ea01b-b933-4f2c-b353-fd10025a75d9" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.380997    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="782e5973-dc08-4e8a-916a-b99deef909c0" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.381010    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1e9aec-c264-4d18-a5af-f930f8fdea14" containerName="istio-proxy"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.381018    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:32.381050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.381028    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:32.387379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.387351    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.405207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.405170    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:30:32.411889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.411857    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c9477c\"/\"kube-root-ca.crt\""
Oct 30 21:30:32.419596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.411949    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c9477c\"/\"openshift-service-ca.crt\""
Oct 30 21:30:32.419596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.412071    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c9477c\"/\"default-dockercfg-5zqjn\""
Oct 30 21:30:32.419596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.416209    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c9477c\"/\"test-model-c9477c-kserve-self-signed-certs\""
Oct 30 21:30:32.448180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448146    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.448338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448306    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.448435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448417    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.448476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448448    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5x6q\" (UniqueName: \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.448542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448485    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.448628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.448600    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.549614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549572    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.549614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549613    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.549904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549815    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.549904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549897    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.550019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549974    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.550019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.549996    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5x6q\" (UniqueName: \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.550526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.550482    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.550656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.550502    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.550932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.550909    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.552359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.552337    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.552557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.552538    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.572068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.572043    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5x6q\" (UniqueName: \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\") pod \"test-model-c9477c-kserve-75ff87dcd5-dvqb6\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") " pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:32.708876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:32.708786    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:33.041137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:33.041109    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:30:33.043027 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:33.042990    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec4ad81_3aa2_4bc5_b6d5_0bdecbf932ec.slice/crio-8e1154ab860fd54e0b73b7a9ca4aeb472b918415c14f92454ab9883a5dab3a26 WatchSource:0}: Error finding container 8e1154ab860fd54e0b73b7a9ca4aeb472b918415c14f92454ab9883a5dab3a26: Status 404 returned error can't find the container with id 8e1154ab860fd54e0b73b7a9ca4aeb472b918415c14f92454ab9883a5dab3a26
Oct 30 21:30:33.353946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:33.353907    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" event={"ID":"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec","Type":"ContainerStarted","Data":"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"}
Oct 30 21:30:33.354106 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:33.353958    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" event={"ID":"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec","Type":"ContainerStarted","Data":"8e1154ab860fd54e0b73b7a9ca4aeb472b918415c14f92454ab9883a5dab3a26"}
Oct 30 21:30:33.420359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:33.420309    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" podStartSLOduration=1.420297372 podStartE2EDuration="1.420297372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:33.420145348 +0000 UTC m=+1776.831684783" watchObservedRunningTime="2026-10-30 21:30:33.420297372 +0000 UTC m=+1776.831836804"
Oct 30 21:30:33.675094 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:30:33.675009    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:30:33.710010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:33.709978    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:42.709201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:42.709168    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:42.723185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:42.723156    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:43.457374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:43.457341    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:30:48.296740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.296705    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:30:48.297717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297698    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:48.297717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297716    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297743    7037 cpu_manager.go:414] "containerMap: removing container" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297749    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297785    7037 cpu_manager.go:414] "containerMap: removing container" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297791    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297813    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:48.297836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297818    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:48.298024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297956    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae54cbf-9e80-4ba6-912a-8e548aeca199" containerName="cleanup"
Oct 30 21:30:48.298024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297967    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c38db6-9750-4843-add2-c44fad28b229" containerName="cleanup"
Oct 30 21:30:48.298024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297974    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="13facc54-bb4c-4a8c-9391-815f5e83aa50" containerName="cleanup"
Oct 30 21:30:48.298024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297983    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c018ac5-609d-42f9-a671-777c071228f7" containerName="cleanup"
Oct 30 21:30:48.298024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.297990    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0483d920-639f-46ae-9d0f-112afb50c5eb" containerName="cleanup"
Oct 30 21:30:48.301953 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.301936    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.341954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.341915    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:30:48.409451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409417    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.409659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409476    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.409659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409597    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.409761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409735    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.409834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409819    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.409878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409852    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.410001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.409986    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.410102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.410089    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.410145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.410134    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn97t\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.511949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.511917    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.511965    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.511998    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512056    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512109    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512131    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512247    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512355    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512382    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512421    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn97t\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512480    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512588    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512788    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.512843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.512794    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.514382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.514359    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.514675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.514657    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.542959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.542929    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.542959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.542941    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn97t\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.620154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.620119    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:48.980207 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:48.980177    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d869b7_969e_4b6b_b0ad_ac247aa8f871.slice/crio-89b508d15d06139e676f8bdebf50525a16b29a4071599b29512479100068aa21 WatchSource:0}: Error finding container 89b508d15d06139e676f8bdebf50525a16b29a4071599b29512479100068aa21: Status 404 returned error can't find the container with id 89b508d15d06139e676f8bdebf50525a16b29a4071599b29512479100068aa21
Oct 30 21:30:48.982706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.982680    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:30:48.983570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.983537    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:48.983651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.983600    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:48.983651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:48.983630    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:30:49.502521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:49.502453    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" event={"ID":"12d869b7-969e-4b6b-b0ad-ac247aa8f871","Type":"ContainerStarted","Data":"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"}
Oct 30 21:30:49.503026 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:49.502557    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" event={"ID":"12d869b7-969e-4b6b-b0ad-ac247aa8f871","Type":"ContainerStarted","Data":"89b508d15d06139e676f8bdebf50525a16b29a4071599b29512479100068aa21"}
Oct 30 21:30:49.563377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:49.563324    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podStartSLOduration=1.56330996 podStartE2EDuration="1.56330996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:49.559624097 +0000 UTC m=+1792.971163533" watchObservedRunningTime="2026-10-30 21:30:49.56330996 +0000 UTC m=+1792.974849396"
Oct 30 21:30:49.622239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:49.620609    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:50.586218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.586180    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:30:50.591456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.591433    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.604674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.604650    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:30:50.608723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.608703    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f23eae\""
Oct 30 21:30:50.609500 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.609473    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.622803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.622779    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 30 21:30:50.622939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.622823    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 30 21:30:50.641199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.641168    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f23eae-dockercfg-cb57h\""
Oct 30 21:30:50.658834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.658794    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:30:50.663219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.663189    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:30:50.680105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.677524    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.680105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.677844    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rch\" (UniqueName: \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.781135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.780938    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.782952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.782921    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rch\" (UniqueName: \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.786648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.786618    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.787712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.787678    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9lw\" (UniqueName: \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.789095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.789059    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.822524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.822287    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rch\" (UniqueName: \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\") pod \"payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.889978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.889888    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9lw\" (UniqueName: \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.890156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.890122    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.891122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.891098    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.908260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.908233    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:50.913421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.913393    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9lw\" (UniqueName: \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\") pod \"payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:50.926819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:50.926789    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:51.305690 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:51.305610    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc15ba92_ed3a_4f5c_9335_029de83660c6.slice/crio-df8600b09419f1c9ab828fbf4c95295b25af496181798eaaf6ebd4d73157236e WatchSource:0}: Error finding container df8600b09419f1c9ab828fbf4c95295b25af496181798eaaf6ebd4d73157236e: Status 404 returned error can't find the container with id df8600b09419f1c9ab828fbf4c95295b25af496181798eaaf6ebd4d73157236e
Oct 30 21:30:51.305978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.305947    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:30:51.310477 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:30:51.310450    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b9c731_713e_44f4_a050_b7ce234e6fa8.slice/crio-c2110296707b5fe53f1096ef4d1bcfb5ae3c2700ef8be17e381b24480ea3f0f3 WatchSource:0}: Error finding container c2110296707b5fe53f1096ef4d1bcfb5ae3c2700ef8be17e381b24480ea3f0f3: Status 404 returned error can't find the container with id c2110296707b5fe53f1096ef4d1bcfb5ae3c2700ef8be17e381b24480ea3f0f3
Oct 30 21:30:51.318542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.316169    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:30:51.520857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.520823    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" event={"ID":"d8b9c731-713e-44f4-a050-b7ce234e6fa8","Type":"ContainerStarted","Data":"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"}
Oct 30 21:30:51.521047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.520884    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" event={"ID":"d8b9c731-713e-44f4-a050-b7ce234e6fa8","Type":"ContainerStarted","Data":"c2110296707b5fe53f1096ef4d1bcfb5ae3c2700ef8be17e381b24480ea3f0f3"}
Oct 30 21:30:51.521047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.520907    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:30:51.523013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.522984    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" event={"ID":"fc15ba92-ed3a-4f5c-9335-029de83660c6","Type":"ContainerStarted","Data":"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"}
Oct 30 21:30:51.523159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.523027    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" event={"ID":"fc15ba92-ed3a-4f5c-9335-029de83660c6","Type":"ContainerStarted","Data":"df8600b09419f1c9ab828fbf4c95295b25af496181798eaaf6ebd4d73157236e"}
Oct 30 21:30:51.523237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.523177    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:30:51.565165 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.565068    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" podStartSLOduration=1.565051747 podStartE2EDuration="1.565051747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:51.561943266 +0000 UTC m=+1794.973482704" watchObservedRunningTime="2026-10-30 21:30:51.565051747 +0000 UTC m=+1794.976591221"
Oct 30 21:30:51.608752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.608698    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" podStartSLOduration=1.608685219 podStartE2EDuration="1.608685219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:51.606913698 +0000 UTC m=+1795.018453133" watchObservedRunningTime="2026-10-30 21:30:51.608685219 +0000 UTC m=+1795.020224679"
Oct 30 21:30:51.621317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.621287    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 30 21:30:51.621458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:51.621333    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 30 21:30:52.620467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:52.620429    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 30 21:30:52.620899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:52.620518    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 30 21:30:53.621049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:53.621021    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 30 21:30:53.621529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:53.621476    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 30 21:30:54.620883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:54.620847    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 30 21:30:54.621056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:54.620901    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 30 21:30:55.625449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:55.625418    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:55.625925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:55.625907    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:30:55.626519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:30:55.626483    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:31:02.556267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:02.556220    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:31:02.557244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:02.557220    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:31:03.674231 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:31:03.674191    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:31:33.673827 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:31:33.673777    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:31:34.501980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.501945    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:31:34.507221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.507201    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.529373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.529343    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f23eae-serving-cert\""
Oct 30 21:31:34.534829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.534797    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:31:34.618275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.618243    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.618467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.618443    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.618606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.618589    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.618655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.618623    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2sp\" (UniqueName: \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.719775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.719741    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.720324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.719821    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.720324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.719846    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2sp\" (UniqueName: \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.720324 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:31:34.719874    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-serving-cert: secret "maas-api-e2e-inf-b-f23eae-serving-cert" not found
Oct 30 21:31:34.720324 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:31:34.719927    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls podName:389efca7-cc35-480d-8e07-d857c497b058 nodeName:}" failed. No retries permitted until 2026-10-30 21:31:35.219912865 +0000 UTC m=+1838.631452302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls") pod "maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" (UID: "389efca7-cc35-480d-8e07-d857c497b058") : secret "maas-api-e2e-inf-b-f23eae-serving-cert" not found
Oct 30 21:31:34.720324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.719995    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.720980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.720959    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.722724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.722701    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:34.750063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:34.750026    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2sp\" (UniqueName: \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:35.229035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:35.229000    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:35.231501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:35.231477    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") pod \"maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:35.424823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:35.424793    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:35.770831 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:31:35.770749    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389efca7_cc35_480d_8e07_d857c497b058.slice/crio-828a426ae628c28015cab12d5b6c4ef5eff43b5d8910169d30aac24a9cc991af WatchSource:0}: Error finding container 828a426ae628c28015cab12d5b6c4ef5eff43b5d8910169d30aac24a9cc991af: Status 404 returned error can't find the container with id 828a426ae628c28015cab12d5b6c4ef5eff43b5d8910169d30aac24a9cc991af
Oct 30 21:31:35.771970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:35.771944    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:31:35.948957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:35.948923    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" event={"ID":"389efca7-cc35-480d-8e07-d857c497b058","Type":"ContainerStarted","Data":"828a426ae628c28015cab12d5b6c4ef5eff43b5d8910169d30aac24a9cc991af"}
Oct 30 21:31:36.957980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:36.957943    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" event={"ID":"389efca7-cc35-480d-8e07-d857c497b058","Type":"ContainerStarted","Data":"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"}
Oct 30 21:31:36.958386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:36.958039    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:31:37.008027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:37.007972    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" podStartSLOduration=2.634874934 podStartE2EDuration="3.007957s" totalImagesPullingTime="373.082066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:37.004463905 +0000 UTC m=+1840.416003342" watchObservedRunningTime="2026-10-30 21:31:37.007957 +0000 UTC m=+1840.419496435"
Oct 30 21:31:42.973020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:31:42.972990    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:32:03.678931 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:32:03.678892    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:32:33.673820 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:32:33.673776    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:33:03.674810 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:33:03.674722    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:33:33.676932 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:33:33.676890    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:33:36.137881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:36.137845    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:33:36.151121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:36.151089    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:33:41.138200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:41.138159    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing" containerID="cri-o://357dffac3d5f6be4b1c73e0062d8cc230b11c68e6376295dbf4f28b55f2ff0a1" gracePeriod=25
Oct 30 21:33:41.151769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:41.151730    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing" containerID="cri-o://8e7a9055dbfaea49d11e677a633cf94da856107539bc2454deb2cc055a1365de" gracePeriod=25
Oct 30 21:33:41.996208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:41.996170    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:41.996208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:41.996193    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:51.995318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:51.995275    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:51.995755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:33:51.995292    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:34:01.365264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.365227    7037 generic.go:356] "Generic (PLEG): container finished" podID="a50f2743-a063-469e-9dba-da4a43ac857a" containerID="8e7a9055dbfaea49d11e677a633cf94da856107539bc2454deb2cc055a1365de" exitCode=0
Oct 30 21:34:01.365741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.365302    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" event={"ID":"a50f2743-a063-469e-9dba-da4a43ac857a","Type":"ContainerDied","Data":"8e7a9055dbfaea49d11e677a633cf94da856107539bc2454deb2cc055a1365de"}
Oct 30 21:34:01.367162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.367140    7037 generic.go:356] "Generic (PLEG): container finished" podID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerID="357dffac3d5f6be4b1c73e0062d8cc230b11c68e6376295dbf4f28b55f2ff0a1" exitCode=0
Oct 30 21:34:01.367372 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.367213    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" event={"ID":"3339715f-18ba-4d60-9f5b-54412d6794e4","Type":"ContainerDied","Data":"357dffac3d5f6be4b1c73e0062d8cc230b11c68e6376295dbf4f28b55f2ff0a1"}
Oct 30 21:34:01.795735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.795705    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:34:01.850810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.850779    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:34:01.894539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.894446    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\" (UniqueName: \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\") pod \"3339715f-18ba-4d60-9f5b-54412d6794e4\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") "
Oct 30 21:34:01.897667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.897628    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\") pod \"3339715f-18ba-4d60-9f5b-54412d6794e4\" (UID: \"3339715f-18ba-4d60-9f5b-54412d6794e4\") "
Oct 30 21:34:01.901769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.897798    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume" pod "3339715f-18ba-4d60-9f5b-54412d6794e4" (UID: "3339715f-18ba-4d60-9f5b-54412d6794e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:01.904437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.904402    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3339715f-18ba-4d60-9f5b-54412d6794e4-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:01.907029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:01.906994    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh" pod "3339715f-18ba-4d60-9f5b-54412d6794e4" (UID: "3339715f-18ba-4d60-9f5b-54412d6794e4"). InnerVolumeSpecName "kube-api-access-pdnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:02.005238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.005152    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\") pod \"a50f2743-a063-469e-9dba-da4a43ac857a\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") "
Oct 30 21:34:02.005238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.005227    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\" (UniqueName: \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\") pod \"a50f2743-a063-469e-9dba-da4a43ac857a\" (UID: \"a50f2743-a063-469e-9dba-da4a43ac857a\") "
Oct 30 21:34:02.005524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.005469    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume" pod "a50f2743-a063-469e-9dba-da4a43ac857a" (UID: "a50f2743-a063-469e-9dba-da4a43ac857a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:02.006137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.006117    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50f2743-a063-469e-9dba-da4a43ac857a-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:02.006179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.006140    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdnwh\" (UniqueName: \"kubernetes.io/projected/3339715f-18ba-4d60-9f5b-54412d6794e4-kube-api-access-pdnwh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:02.007327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.007309    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h" pod "a50f2743-a063-469e-9dba-da4a43ac857a" (UID: "a50f2743-a063-469e-9dba-da4a43ac857a"). InnerVolumeSpecName "kube-api-access-vm84h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:02.107987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.107926    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm84h\" (UniqueName: \"kubernetes.io/projected/a50f2743-a063-469e-9dba-da4a43ac857a-kube-api-access-vm84h\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:02.376506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.376469    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"
Oct 30 21:34:02.376994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.376469    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889" event={"ID":"a50f2743-a063-469e-9dba-da4a43ac857a","Type":"ContainerDied","Data":"d17550b7856613fce588b20448d2e1199ee7e561354bf2e72d501c608ebb1610"}
Oct 30 21:34:02.376994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.376601    7037 scope.go:122] "RemoveContainer" containerID="8e7a9055dbfaea49d11e677a633cf94da856107539bc2454deb2cc055a1365de"
Oct 30 21:34:02.378291 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.378263    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp" event={"ID":"3339715f-18ba-4d60-9f5b-54412d6794e4","Type":"ContainerDied","Data":"79a5bcc324490cd7ffefd927fccdcc42da2da5040a28370a2088c519a3a51444"}
Oct 30 21:34:02.378395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.378293    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"
Oct 30 21:34:02.388907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.388886    7037 scope.go:122] "RemoveContainer" containerID="357dffac3d5f6be4b1c73e0062d8cc230b11c68e6376295dbf4f28b55f2ff0a1"
Oct 30 21:34:02.431228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.431199    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:34:02.442741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.442712    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31999-74bcfbbd68-j77lp"]
Oct 30 21:34:02.467033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.466974    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:34:02.470845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.470818    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31999-57f7b8f867-8z889"]
Oct 30 21:34:02.711190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.711096    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" path="/var/lib/kubelet/pods/3339715f-18ba-4d60-9f5b-54412d6794e4/volumes"
Oct 30 21:34:02.711746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:02.711723    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" path="/var/lib/kubelet/pods/a50f2743-a063-469e-9dba-da4a43ac857a/volumes"
Oct 30 21:34:03.712664 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:34:03.712621    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:34:07.348829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:07.348739    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:34:07.349385 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:07.348997    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy" containerID="cri-o://e4a5c5ab00442ead7ac53d3c7c740d4af954919ea5740c2b044a915a56e5d1e7" gracePeriod=30
Oct 30 21:34:07.702211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:07.702135    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:34:07.702415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:07.702370    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" podUID="c706a8c9-0ed3-4c71-9d21-556293c78063" containerName="maas-api" containerID="cri-o://c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93" gracePeriod=30
Oct 30 21:34:08.059870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.059841    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:34:08.112528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.110022    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\") pod \"c706a8c9-0ed3-4c71-9d21-556293c78063\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") "
Oct 30 21:34:08.112528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.110129    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\") pod \"c706a8c9-0ed3-4c71-9d21-556293c78063\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") "
Oct 30 21:34:08.112528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.110226    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\") pod \"c706a8c9-0ed3-4c71-9d21-556293c78063\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") "
Oct 30 21:34:08.112528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.110547    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\" (UniqueName: \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\") pod \"c706a8c9-0ed3-4c71-9d21-556293c78063\" (UID: \"c706a8c9-0ed3-4c71-9d21-556293c78063\") "
Oct 30 21:34:08.112955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.112918    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca" pod "c706a8c9-0ed3-4c71-9d21-556293c78063" (UID: "c706a8c9-0ed3-4c71-9d21-556293c78063"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:08.121562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.121027    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs" pod "c706a8c9-0ed3-4c71-9d21-556293c78063" (UID: "c706a8c9-0ed3-4c71-9d21-556293c78063"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:08.121852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.121825    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz" pod "c706a8c9-0ed3-4c71-9d21-556293c78063" (UID: "c706a8c9-0ed3-4c71-9d21-556293c78063"). InnerVolumeSpecName "kube-api-access-p5dpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:08.122995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.122966    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls" pod "c706a8c9-0ed3-4c71-9d21-556293c78063" (UID: "c706a8c9-0ed3-4c71-9d21-556293c78063"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:08.213593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.213540    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5dpz\" (UniqueName: \"kubernetes.io/projected/c706a8c9-0ed3-4c71-9d21-556293c78063-kube-api-access-p5dpz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:08.213593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.213576    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c706a8c9-0ed3-4c71-9d21-556293c78063-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:08.213593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.213587    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:08.213593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.213596    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c706a8c9-0ed3-4c71-9d21-556293c78063-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:08.458635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.458590    7037 generic.go:356] "Generic (PLEG): container finished" podID="c706a8c9-0ed3-4c71-9d21-556293c78063" containerID="c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93" exitCode=0
Oct 30 21:34:08.459082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.459045    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" event={"ID":"c706a8c9-0ed3-4c71-9d21-556293c78063","Type":"ContainerDied","Data":"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"}
Oct 30 21:34:08.459150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.459133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8" event={"ID":"c706a8c9-0ed3-4c71-9d21-556293c78063","Type":"ContainerDied","Data":"2fb9192f4d7368789e3c4fdfe8ab8b1a7560ad589ef0d85f8afd8e9ed3d63965"}
Oct 30 21:34:08.459197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.459170    7037 scope.go:122] "RemoveContainer" containerID="c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"
Oct 30 21:34:08.459351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.459336    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"
Oct 30 21:34:08.482796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.482766    7037 scope.go:122] "RemoveContainer" containerID="c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"
Oct 30 21:34:08.483102 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:34:08.483081    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93\": container with ID starting with c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93 not found: ID does not exist" containerID="c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"
Oct 30 21:34:08.483161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.483111    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93"} err="failed to get container status \"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93\": rpc error: code = NotFound desc = could not find container \"c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93\": container with ID starting with c711974cc11b1de554ff3fc9115209c21e877a91539981e1218c3b4718804a93 not found: ID does not exist"
Oct 30 21:34:08.521055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.521025    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:34:08.524824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.524797    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31999-5f44957c74-tr5x8"]
Oct 30 21:34:08.711385 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:08.711311    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c706a8c9-0ed3-4c71-9d21-556293c78063" path="/var/lib/kubelet/pods/c706a8c9-0ed3-4c71-9d21-556293c78063/volumes"
Oct 30 21:34:12.493526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.493477    7037 generic.go:356] "Generic (PLEG): container finished" podID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerID="e4a5c5ab00442ead7ac53d3c7c740d4af954919ea5740c2b044a915a56e5d1e7" exitCode=0
Oct 30 21:34:12.493526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.493520    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" event={"ID":"4ae3eda8-0abe-45c3-ae7e-59bd41848798","Type":"ContainerDied","Data":"e4a5c5ab00442ead7ac53d3c7c740d4af954919ea5740c2b044a915a56e5d1e7"}
Oct 30 21:34:12.636378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.636314    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:34:12.704424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704359    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704468    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704564    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704635    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704675    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:12.704832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704708    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704727    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:12.704832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704755    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.704832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704811    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.705115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704861    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.705115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704918    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:12.705115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.704976    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:12.705115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705043    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\") pod \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\" (UID: \"4ae3eda8-0abe-45c3-ae7e-59bd41848798\") "
Oct 30 21:34:12.705392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705210    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:12.705795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705765    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.705795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705794    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.705979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705807    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.705979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705820    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.705979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.705832    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.707897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.707861    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:12.707897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.707872    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "kube-api-access-sls9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:12.707897 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.707886    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:12.708137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.708012    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token" pod "4ae3eda8-0abe-45c3-ae7e-59bd41848798" (UID: "4ae3eda8-0abe-45c3-ae7e-59bd41848798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:12.808012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.807980    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.808012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.808012    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sls9z\" (UniqueName: \"kubernetes.io/projected/4ae3eda8-0abe-45c3-ae7e-59bd41848798-kube-api-access-sls9z\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.808228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.808027    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:12.808228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:12.808036    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae3eda8-0abe-45c3-ae7e-59bd41848798-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:13.503656 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:13.503626    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"
Oct 30 21:34:13.504127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:13.503658    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg" event={"ID":"4ae3eda8-0abe-45c3-ae7e-59bd41848798","Type":"ContainerDied","Data":"98ed41c340b90bd0ced46ebbe422224062b05d4aeba8e346b720636b0481db2d"}
Oct 30 21:34:13.504127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:13.503723    7037 scope.go:122] "RemoveContainer" containerID="e4a5c5ab00442ead7ac53d3c7c740d4af954919ea5740c2b044a915a56e5d1e7"
Oct 30 21:34:13.571148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:13.571114    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:34:13.582684 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:13.582651    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-lkltg"]
Oct 30 21:34:14.712300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:14.712258    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" path="/var/lib/kubelet/pods/4ae3eda8-0abe-45c3-ae7e-59bd41848798/volumes"
Oct 30 21:34:15.918552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:15.918514    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:34:15.919047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:15.918785    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy" containerID="cri-o://3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8" gracePeriod=30
Oct 30 21:34:17.638020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:17.637985    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:34:17.638447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:17.638174    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerName="discovery" containerID="cri-o://8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c" gracePeriod=30
Oct 30 21:34:17.944465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:17.944439    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:34:18.093944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.093912    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.093966    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094085    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094146    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094191    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094438    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094513    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\") pod \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\" (UID: \"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56\") "
Oct 30 21:34:18.094772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.094661    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:18.095271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.095245    7037 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-ca-configmap\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.096593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.096565    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:18.097344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.096752    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:18.097344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.096777    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "kube-api-access-jr8zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:18.097344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.097129    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:18.100368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.100327    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:18.101082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.101038    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token" pod "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" (UID: "6d62e254-0c7f-48fc-a9aa-be64c0b8bc56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197042    7037 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-kubeconfig\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197073    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8zj\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-kube-api-access-jr8zj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197084    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197092    7037 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-istio-csr-dns-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197102    7037 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-cacerts\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.197119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.197110    7037 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56-local-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:18.555677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.555584    7037 generic.go:356] "Generic (PLEG): container finished" podID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerID="8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c" exitCode=0
Oct 30 21:34:18.555843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.555753    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" event={"ID":"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56","Type":"ContainerDied","Data":"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"}
Oct 30 21:34:18.555843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.555812    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"
Oct 30 21:34:18.555971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.555841    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx" event={"ID":"6d62e254-0c7f-48fc-a9aa-be64c0b8bc56","Type":"ContainerDied","Data":"342c7065d19ff800d062375f75e960cfe7909f36b6fc8582426085fe36a43005"}
Oct 30 21:34:18.555971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.555879    7037 scope.go:122] "RemoveContainer" containerID="8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"
Oct 30 21:34:18.568966 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.568946    7037 scope.go:122] "RemoveContainer" containerID="8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"
Oct 30 21:34:18.569245 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:34:18.569225    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c\": container with ID starting with 8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c not found: ID does not exist" containerID="8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"
Oct 30 21:34:18.569286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.569253    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c"} err="failed to get container status \"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c\": rpc error: code = NotFound desc = could not find container \"8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c\": container with ID starting with 8a30be13cbe79ac0fe0c590b3901bfb86f4953fd0fb8c675c1f5669ab551f17c not found: ID does not exist"
Oct 30 21:34:18.610640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.610612    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:34:18.622725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.622693    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dtvjx"]
Oct 30 21:34:18.712961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:18.712927    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" path="/var/lib/kubelet/pods/6d62e254-0c7f-48fc-a9aa-be64c0b8bc56/volumes"
Oct 30 21:34:21.241913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.241886    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:34:21.244292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244259    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244308    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244442    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244569    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244612    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244614    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244643    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244681    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244828    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244888    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\") pod \"6b921d44-4189-41c2-94c0-8cc915a00dde\" (UID: \"6b921d44-4189-41c2-94c0-8cc915a00dde\") "
Oct 30 21:34:21.244913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.244893    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:21.245462 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.245441    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:21.245876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.245732    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.245876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.245753    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b921d44-4189-41c2-94c0-8cc915a00dde-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.245876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.245789    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.245876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.245828    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:21.246286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.246262    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:21.247565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.247538    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:21.257521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.248350    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:21.257521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.248449    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:21.257521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.248805    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd" pod "6b921d44-4189-41c2-94c0-8cc915a00dde" (UID: "6b921d44-4189-41c2-94c0-8cc915a00dde"). InnerVolumeSpecName "kube-api-access-d7jqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350177    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350222    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350236    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350248    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350262    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jqd\" (UniqueName: \"kubernetes.io/projected/6b921d44-4189-41c2-94c0-8cc915a00dde-kube-api-access-d7jqd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.350390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.350275    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b921d44-4189-41c2-94c0-8cc915a00dde-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.586195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.585590    7037 generic.go:356] "Generic (PLEG): container finished" podID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerID="3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8" exitCode=0
Oct 30 21:34:21.586195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.585666    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" event={"ID":"6b921d44-4189-41c2-94c0-8cc915a00dde","Type":"ContainerDied","Data":"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"}
Oct 30 21:34:21.586195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.585770    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs" event={"ID":"6b921d44-4189-41c2-94c0-8cc915a00dde","Type":"ContainerDied","Data":"f00b440703e4a354379c56e17655c9b5e69cdf644296ccfb015723c8042adc54"}
Oct 30 21:34:21.586195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.585804    7037 scope.go:122] "RemoveContainer" containerID="3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"
Oct 30 21:34:21.586195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.585930    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"
Oct 30 21:34:21.609327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.609306    7037 scope.go:122] "RemoveContainer" containerID="3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"
Oct 30 21:34:21.609777 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:34:21.609739    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8\": container with ID starting with 3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8 not found: ID does not exist" containerID="3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"
Oct 30 21:34:21.609894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.609774    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8"} err="failed to get container status \"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8\": rpc error: code = NotFound desc = could not find container \"3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8\": container with ID starting with 3a1315960268e4cb0e89e06498ea2aee25cad12268889d5591b03572f91355b8 not found: ID does not exist"
Oct 30 21:34:21.641751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.641716    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:34:21.653655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:21.653626    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-lxzhs"]
Oct 30 21:34:22.709939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:34:22.709905    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" path="/var/lib/kubelet/pods/6b921d44-4189-41c2-94c0-8cc915a00dde/volumes"
Oct 30 21:34:33.673173 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:34:33.673134    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:35:00.273834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.273799    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:35:00.274349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.274020    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" podUID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" containerName="maas-api" containerID="cri-o://8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030" gracePeriod=30
Oct 30 21:35:00.579755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.579725    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:35:00.647134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.647089    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\") pod \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") "
Oct 30 21:35:00.647338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.647153    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\") pod \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") "
Oct 30 21:35:00.647338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.647239    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\" (UniqueName: \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\") pod \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") "
Oct 30 21:35:00.647470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.647337    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\") pod \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\" (UID: \"d4fe1de9-7a76-4845-bf18-96528a9df3d1\") "
Oct 30 21:35:00.648915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.648882    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca" pod "d4fe1de9-7a76-4845-bf18-96528a9df3d1" (UID: "d4fe1de9-7a76-4845-bf18-96528a9df3d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:00.651149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.651124    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls" pod "d4fe1de9-7a76-4845-bf18-96528a9df3d1" (UID: "d4fe1de9-7a76-4845-bf18-96528a9df3d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:00.651256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.651193    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs" pod "d4fe1de9-7a76-4845-bf18-96528a9df3d1" (UID: "d4fe1de9-7a76-4845-bf18-96528a9df3d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:00.659700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.659668    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf" pod "d4fe1de9-7a76-4845-bf18-96528a9df3d1" (UID: "d4fe1de9-7a76-4845-bf18-96528a9df3d1"). InnerVolumeSpecName "kube-api-access-42tpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:00.749734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.749695    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:00.749734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.749728    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fe1de9-7a76-4845-bf18-96528a9df3d1-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:00.749734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.749741    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tpf\" (UniqueName: \"kubernetes.io/projected/d4fe1de9-7a76-4845-bf18-96528a9df3d1-kube-api-access-42tpf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:00.750062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.749755    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fe1de9-7a76-4845-bf18-96528a9df3d1-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:00.901883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.901845    7037 generic.go:356] "Generic (PLEG): container finished" podID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" containerID="8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030" exitCode=0
Oct 30 21:35:00.902095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.901893    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"
Oct 30 21:35:00.902095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.901931    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" event={"ID":"d4fe1de9-7a76-4845-bf18-96528a9df3d1","Type":"ContainerDied","Data":"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"}
Oct 30 21:35:00.902095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.901989    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb" event={"ID":"d4fe1de9-7a76-4845-bf18-96528a9df3d1","Type":"ContainerDied","Data":"c2a739f1ee285dab60de670e7ce45cc520b0182b9e0a8e9c950ab193dae000f0"}
Oct 30 21:35:00.902095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.902016    7037 scope.go:122] "RemoveContainer" containerID="8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"
Oct 30 21:35:00.921766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.921740    7037 scope.go:122] "RemoveContainer" containerID="8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"
Oct 30 21:35:00.922252 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:35:00.922145    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030\": container with ID starting with 8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030 not found: ID does not exist" containerID="8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"
Oct 30 21:35:00.922252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.922188    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030"} err="failed to get container status \"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030\": rpc error: code = NotFound desc = could not find container \"8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030\": container with ID starting with 8ff9b053061fce11d6026e04a7b6e2d3d1d7ba085ee1658febc874662a40a030 not found: ID does not exist"
Oct 30 21:35:00.955828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.955794    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:35:00.975531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:00.974459    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-093daa-6ff7db8669-r6mtb"]
Oct 30 21:35:02.709571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:02.709535    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" path="/var/lib/kubelet/pods/d4fe1de9-7a76-4845-bf18-96528a9df3d1/volumes"
Oct 30 21:35:03.673584 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:35:03.673530    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:35:33.674343 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:35:33.674302    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:35:35.245284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.245252    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:35:35.246171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246152    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246173    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246185    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerName="discovery"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246190    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerName="discovery"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246199    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246204    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246212    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" containerName="maas-api"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246217    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" containerName="maas-api"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246253    7037 cpu_manager.go:414] "containerMap: removing container" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy"
Oct 30 21:35:35.246256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246258    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246268    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c706a8c9-0ed3-4c71-9d21-556293c78063" containerName="maas-api"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246273    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c706a8c9-0ed3-4c71-9d21-556293c78063" containerName="maas-api"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246282    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246286    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246413    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3339715f-18ba-4d60-9f5b-54412d6794e4" containerName="payload-processing"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246420    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d62e254-0c7f-48fc-a9aa-be64c0b8bc56" containerName="discovery"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246428    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50f2743-a063-469e-9dba-da4a43ac857a" containerName="payload-pre-processing"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246435    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fe1de9-7a76-4845-bf18-96528a9df3d1" containerName="maas-api"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246448    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae3eda8-0abe-45c3-ae7e-59bd41848798" containerName="istio-proxy"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246454    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c706a8c9-0ed3-4c71-9d21-556293c78063" containerName="maas-api"
Oct 30 21:35:35.246639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.246461    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b921d44-4189-41c2-94c0-8cc915a00dde" containerName="istio-proxy"
Oct 30 21:35:35.249877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.249861    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.281400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281365    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.281651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281446    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.281651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281587    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.282302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281659    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.282302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281740    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.282302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.281899    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpm7b\" (UniqueName: \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.296651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.296621    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a4782a\"/\"default-dockercfg-w8w5r\""
Oct 30 21:35:35.296651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.296618    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a4782a\"/\"auth-test-model-a4782a-kserve-self-signed-certs\""
Oct 30 21:35:35.296874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.296652    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a4782a\"/\"openshift-service-ca.crt\""
Oct 30 21:35:35.296874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.296635    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a4782a\"/\"kube-root-ca.crt\""
Oct 30 21:35:35.301121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.301094    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:35:35.383225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383195    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpm7b\" (UniqueName: \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383247    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383281    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383346    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383373    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383417    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383768    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383827    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.383934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.383834    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.385666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.385622    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.385929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.385911    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.418344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.418315    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpm7b\" (UniqueName: \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\") pod \"auth-test-model-a4782a-kserve-559f6b6f65-pn5wx\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") " pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.563153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.563072    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:35.948023 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:35:35.947981    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91434dc0_216a_4170_83a0_21e86c3e963c.slice/crio-3d9af9c5c858da5d30f7beb0fc5e8269f45612578bb1925c6e659f51106e79e3 WatchSource:0}: Error finding container 3d9af9c5c858da5d30f7beb0fc5e8269f45612578bb1925c6e659f51106e79e3: Status 404 returned error can't find the container with id 3d9af9c5c858da5d30f7beb0fc5e8269f45612578bb1925c6e659f51106e79e3
Oct 30 21:35:35.949792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:35.949766    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:35:36.238385 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:36.238294    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" event={"ID":"91434dc0-216a-4170-83a0-21e86c3e963c","Type":"ContainerStarted","Data":"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"}
Oct 30 21:35:36.238385 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:36.238335    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" event={"ID":"91434dc0-216a-4170-83a0-21e86c3e963c","Type":"ContainerStarted","Data":"3d9af9c5c858da5d30f7beb0fc5e8269f45612578bb1925c6e659f51106e79e3"}
Oct 30 21:35:36.287815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:36.287764    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" podStartSLOduration=1.287749933 podStartE2EDuration="1.287749933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:36.285743441 +0000 UTC m=+2079.697282869" watchObservedRunningTime="2026-10-30 21:35:36.287749933 +0000 UTC m=+2079.699289404"
Oct 30 21:35:36.563716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:36.563634    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:42.908519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:42.908457    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:35:45.563353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:45.563310    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:45.576259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:45.576236    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:46.334334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:46.334304    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:35:47.909786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:47.909720    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" podUID="5def4fe2-57fc-4b77-a717-e7fd30197625" containerName="payload-processing" containerID="cri-o://015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41" gracePeriod=25
Oct 30 21:35:48.223827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.223802    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:35:48.306510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.306459    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\" (UniqueName: \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\") pod \"5def4fe2-57fc-4b77-a717-e7fd30197625\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") "
Oct 30 21:35:48.306705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.306646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\") pod \"5def4fe2-57fc-4b77-a717-e7fd30197625\" (UID: \"5def4fe2-57fc-4b77-a717-e7fd30197625\") "
Oct 30 21:35:48.307110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.307079    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume" pod "5def4fe2-57fc-4b77-a717-e7fd30197625" (UID: "5def4fe2-57fc-4b77-a717-e7fd30197625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:48.307584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.307549    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5def4fe2-57fc-4b77-a717-e7fd30197625-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:48.308804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.308773    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp" pod "5def4fe2-57fc-4b77-a717-e7fd30197625" (UID: "5def4fe2-57fc-4b77-a717-e7fd30197625"). InnerVolumeSpecName "kube-api-access-wzmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:48.340622 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.340593    7037 generic.go:356] "Generic (PLEG): container finished" podID="5def4fe2-57fc-4b77-a717-e7fd30197625" containerID="015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41" exitCode=0
Oct 30 21:35:48.340749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.340635    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"
Oct 30 21:35:48.340749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.340671    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" event={"ID":"5def4fe2-57fc-4b77-a717-e7fd30197625","Type":"ContainerDied","Data":"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"}
Oct 30 21:35:48.340749 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.340743    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5" event={"ID":"5def4fe2-57fc-4b77-a717-e7fd30197625","Type":"ContainerDied","Data":"8f34b93318ad937205effa5da97c9ba001baf90b012389e2b24e583a5d1257e2"}
Oct 30 21:35:48.340913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.340766    7037 scope.go:122] "RemoveContainer" containerID="015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"
Oct 30 21:35:48.352363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.352340    7037 scope.go:122] "RemoveContainer" containerID="015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"
Oct 30 21:35:48.352649 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:35:48.352629    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41\": container with ID starting with 015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41 not found: ID does not exist" containerID="015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"
Oct 30 21:35:48.352718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.352660    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41"} err="failed to get container status \"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41\": rpc error: code = NotFound desc = could not find container \"015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41\": container with ID starting with 015f24320d64f0b9b1ee65b1f5e20b8cb9fe5165b868e9cb4738faa2bc306f41 not found: ID does not exist"
Oct 30 21:35:48.396449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.396418    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:35:48.407941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.407911    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1eab65-8576d668f6-2lxn5"]
Oct 30 21:35:48.409104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.409076    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmbp\" (UniqueName: \"kubernetes.io/projected/5def4fe2-57fc-4b77-a717-e7fd30197625-kube-api-access-wzmbp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:35:48.711740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:35:48.711709    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5def4fe2-57fc-4b77-a717-e7fd30197625" path="/var/lib/kubelet/pods/5def4fe2-57fc-4b77-a717-e7fd30197625/volumes"
Oct 30 21:36:03.675095 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:03.675040    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:36:26.939221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:26.939182    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:36:26.973881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:26.973846    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:36:28.608643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:28.608605    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:36:28.609136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:28.608858    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" podUID="a050fd78-0321-42a2-8f58-41c0eb50c70b" containerName="maas-api" containerID="cri-o://e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683" gracePeriod=30
Oct 30 21:36:28.950308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:28.950278    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:36:29.077193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.077152    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\") pod \"a050fd78-0321-42a2-8f58-41c0eb50c70b\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") "
Oct 30 21:36:29.077404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.077376    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\") pod \"a050fd78-0321-42a2-8f58-41c0eb50c70b\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") "
Oct 30 21:36:29.077469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.077437    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\") pod \"a050fd78-0321-42a2-8f58-41c0eb50c70b\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") "
Oct 30 21:36:29.077680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.077660    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\" (UniqueName: \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\") pod \"a050fd78-0321-42a2-8f58-41c0eb50c70b\" (UID: \"a050fd78-0321-42a2-8f58-41c0eb50c70b\") "
Oct 30 21:36:29.078392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.078367    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca" pod "a050fd78-0321-42a2-8f58-41c0eb50c70b" (UID: "a050fd78-0321-42a2-8f58-41c0eb50c70b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:29.080920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.080891    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr" pod "a050fd78-0321-42a2-8f58-41c0eb50c70b" (UID: "a050fd78-0321-42a2-8f58-41c0eb50c70b"). InnerVolumeSpecName "kube-api-access-48zmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:29.089631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.089560    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs" pod "a050fd78-0321-42a2-8f58-41c0eb50c70b" (UID: "a050fd78-0321-42a2-8f58-41c0eb50c70b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:36:29.091931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.091887    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls" pod "a050fd78-0321-42a2-8f58-41c0eb50c70b" (UID: "a050fd78-0321-42a2-8f58-41c0eb50c70b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:36:29.182655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.180354    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48zmr\" (UniqueName: \"kubernetes.io/projected/a050fd78-0321-42a2-8f58-41c0eb50c70b-kube-api-access-48zmr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:29.182655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.180387    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:29.182655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.180397    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a050fd78-0321-42a2-8f58-41c0eb50c70b-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:29.182655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.180405    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a050fd78-0321-42a2-8f58-41c0eb50c70b-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:29.711004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.710969    7037 generic.go:356] "Generic (PLEG): container finished" podID="a050fd78-0321-42a2-8f58-41c0eb50c70b" containerID="e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683" exitCode=0
Oct 30 21:36:29.711452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.711024    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"
Oct 30 21:36:29.711452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.711050    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" event={"ID":"a050fd78-0321-42a2-8f58-41c0eb50c70b","Type":"ContainerDied","Data":"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"}
Oct 30 21:36:29.711452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.711108    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x" event={"ID":"a050fd78-0321-42a2-8f58-41c0eb50c70b","Type":"ContainerDied","Data":"bcdcef20aa7852d62264571ddd72b59c91bbc98116861dcab40ce0ae6a1f0bde"}
Oct 30 21:36:29.711452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.711137    7037 scope.go:122] "RemoveContainer" containerID="e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"
Oct 30 21:36:29.723250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.723220    7037 scope.go:122] "RemoveContainer" containerID="e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"
Oct 30 21:36:29.723624 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:29.723602    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683\": container with ID starting with e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683 not found: ID does not exist" containerID="e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"
Oct 30 21:36:29.723687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.723633    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683"} err="failed to get container status \"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683\": rpc error: code = NotFound desc = could not find container \"e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683\": container with ID starting with e4191ae051b1a6cebcefd3711d3dae321f1c6b7fa39fbb007ecad5f07a01b683 not found: ID does not exist"
Oct 30 21:36:29.776132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.776096    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:36:29.786195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:29.786156    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1eab65-75574d5768-t6f6x"]
Oct 30 21:36:30.709846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:30.709814    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a050fd78-0321-42a2-8f58-41c0eb50c70b" path="/var/lib/kubelet/pods/a050fd78-0321-42a2-8f58-41c0eb50c70b/volumes"
Oct 30 21:36:31.939855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:31.939812    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing" containerID="cri-o://cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b" gracePeriod=25
Oct 30 21:36:31.974582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:31.974526    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing" containerID="cri-o://2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e" gracePeriod=25
Oct 30 21:36:33.674365 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:33.674329    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:36:34.768559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.768519    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:36:34.769988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.769960    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5def4fe2-57fc-4b77-a717-e7fd30197625" containerName="payload-processing"
Oct 30 21:36:34.769988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.769988    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5def4fe2-57fc-4b77-a717-e7fd30197625" containerName="payload-processing"
Oct 30 21:36:34.770235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.770089    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a050fd78-0321-42a2-8f58-41c0eb50c70b" containerName="maas-api"
Oct 30 21:36:34.770235 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.770099    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a050fd78-0321-42a2-8f58-41c0eb50c70b" containerName="maas-api"
Oct 30 21:36:34.770360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.770316    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a050fd78-0321-42a2-8f58-41c0eb50c70b" containerName="maas-api"
Oct 30 21:36:34.770360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.770340    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5def4fe2-57fc-4b77-a717-e7fd30197625" containerName="payload-processing"
Oct 30 21:36:34.777167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.777142    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.794307 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.794277    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:36:34.802010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.801980    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-75c17d\"/\"kube-root-ca.crt\""
Oct 30 21:36:34.802010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.801980    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-75c17d\"/\"openshift-service-ca.crt\""
Oct 30 21:36:34.802548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.802526    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-75c17d\"/\"auth-test-model-75c17d-kserve-self-signed-certs\""
Oct 30 21:36:34.802548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.802535    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-75c17d\"/\"default-dockercfg-hmtdc\""
Oct 30 21:36:34.871716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.871677    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.871890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.871751    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.871890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.871876    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.872019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.871947    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.872019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.872015    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqh9h\" (UniqueName: \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.872094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.872061    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973076    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973156    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973222    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqh9h\" (UniqueName: \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973278    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973466    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973555    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973567    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973624    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.973752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.973712    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.975852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.975827    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.976218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.976197    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:34.993683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:34.993658    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqh9h\" (UniqueName: \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\") pod \"auth-test-model-75c17d-kserve-66f8774769-jnh5h\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") " pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:35.094530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:35.093801    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:35.437878 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:36:35.437844    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda32a347f_36a4_4905_a315_ce0d6aebf5f9.slice/crio-d66cd7e57975014d71e2e633033c1a521178a9a1e0931508b59799495cc6f068 WatchSource:0}: Error finding container d66cd7e57975014d71e2e633033c1a521178a9a1e0931508b59799495cc6f068: Status 404 returned error can't find the container with id d66cd7e57975014d71e2e633033c1a521178a9a1e0931508b59799495cc6f068
Oct 30 21:36:35.438867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:35.438836    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:36:35.776446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:35.776343    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" event={"ID":"a32a347f-36a4-4905-a315-ce0d6aebf5f9","Type":"ContainerStarted","Data":"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"}
Oct 30 21:36:35.776446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:35.776397    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" event={"ID":"a32a347f-36a4-4905-a315-ce0d6aebf5f9","Type":"ContainerStarted","Data":"d66cd7e57975014d71e2e633033c1a521178a9a1e0931508b59799495cc6f068"}
Oct 30 21:36:35.844624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:35.844574    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" podStartSLOduration=1.844560086 podStartE2EDuration="1.844560086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:35.840796326 +0000 UTC m=+2139.252335764" watchObservedRunningTime="2026-10-30 21:36:35.844560086 +0000 UTC m=+2139.256099520"
Oct 30 21:36:36.102870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:36.102839    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:36.910394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:36.910351    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:36:36.910394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:36.910379    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:36:38.009203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:38.009164    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:36:38.010112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:38.009426    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy" containerID="cri-o://e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48" gracePeriod=30
Oct 30 21:36:43.293524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.293426    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:36:43.412047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412016    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412126    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412336    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412403    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412431    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412469    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412554    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412592    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.412768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.412620    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\") pod \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\" (UID: \"9c297a8b-24b5-4179-a5ba-3b5ec90c851f\") "
Oct 30 21:36:43.413663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.413633    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:43.413880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.413856    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:43.414591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.414372    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:43.414591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.414553    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:43.414917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.414886    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:43.415104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.415080    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:43.415424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.415402    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:43.416201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.416173    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "kube-api-access-hxlzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:43.417240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.417218    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo" pod "9c297a8b-24b5-4179-a5ba-3b5ec90c851f" (UID: "9c297a8b-24b5-4179-a5ba-3b5ec90c851f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:43.515374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515337    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515370    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515379    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515387    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515396    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515404    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515411    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515422    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxlzd\" (UniqueName: \"kubernetes.io/projected/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-kube-api-access-hxlzd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.515648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.515432    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c297a8b-24b5-4179-a5ba-3b5ec90c851f-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:43.850718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.850678    7037 generic.go:356] "Generic (PLEG): container finished" podID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerID="e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48" exitCode=0
Oct 30 21:36:43.850878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.850733    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"
Oct 30 21:36:43.850878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.850770    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" event={"ID":"9c297a8b-24b5-4179-a5ba-3b5ec90c851f","Type":"ContainerDied","Data":"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"}
Oct 30 21:36:43.850878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.850832    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d" event={"ID":"9c297a8b-24b5-4179-a5ba-3b5ec90c851f","Type":"ContainerDied","Data":"193c42fa839b5ad5e34a997d921994cc654065bae1f46672f377a95f587b04f7"}
Oct 30 21:36:43.850878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.850856    7037 scope.go:122] "RemoveContainer" containerID="e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"
Oct 30 21:36:43.863665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.863642    7037 scope.go:122] "RemoveContainer" containerID="e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"
Oct 30 21:36:43.863967 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:43.863947    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48\": container with ID starting with e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48 not found: ID does not exist" containerID="e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"
Oct 30 21:36:43.864008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.863977    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48"} err="failed to get container status \"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48\": rpc error: code = NotFound desc = could not find container \"e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48\": container with ID starting with e832964fc7a903c487dcb8b71c598fac04576d4b7d03966c0bd12123d8c11f48 not found: ID does not exist"
Oct 30 21:36:43.923798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.923766    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:36:43.928581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:43.928554    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4zm2d"]
Oct 30 21:36:44.710228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:44.710192    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" path="/var/lib/kubelet/pods/9c297a8b-24b5-4179-a5ba-3b5ec90c851f/volumes"
Oct 30 21:36:45.103746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:45.103713    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:45.116317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:45.116288    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:45.883740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:45.883711    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:36:46.909607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:46.909569    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:36:46.910341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:46.910311    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:36:50.294990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:50.294954    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:36:50.295433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:50.295174    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" containerID="cri-o://1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1" gracePeriod=30
Oct 30 21:36:52.305949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.305514    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:36:52.415567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.414324    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\") pod \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") "
Oct 30 21:36:52.415567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.414574    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\" (UniqueName: \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\") pod \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\" (UID: \"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c\") "
Oct 30 21:36:52.466173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.466127    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume" pod "49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" (UID: "49eb04a7-c42c-4eb7-9f58-a94d1e345e2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:52.467557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.467484    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg" pod "49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" (UID: "49eb04a7-c42c-4eb7-9f58-a94d1e345e2c"). InnerVolumeSpecName "kube-api-access-mfwrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:52.482677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.482643    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:36:52.516898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.516812    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:52.516898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.516845    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwrg\" (UniqueName: \"kubernetes.io/projected/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c-kube-api-access-mfwrg\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:52.617640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.617600    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\") pod \"766bc989-145d-42c9-b2d9-4916f69186c6\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") "
Oct 30 21:36:52.617824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.617674    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\" (UniqueName: \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\") pod \"766bc989-145d-42c9-b2d9-4916f69186c6\" (UID: \"766bc989-145d-42c9-b2d9-4916f69186c6\") "
Oct 30 21:36:52.619540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.618169    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume" pod "766bc989-145d-42c9-b2d9-4916f69186c6" (UID: "766bc989-145d-42c9-b2d9-4916f69186c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:52.620212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.620186    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf" pod "766bc989-145d-42c9-b2d9-4916f69186c6" (UID: "766bc989-145d-42c9-b2d9-4916f69186c6"). InnerVolumeSpecName "kube-api-access-hx2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:52.720082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.720036    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766bc989-145d-42c9-b2d9-4916f69186c6-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:52.720082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.720076    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx2mf\" (UniqueName: \"kubernetes.io/projected/766bc989-145d-42c9-b2d9-4916f69186c6-kube-api-access-hx2mf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:52.953383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.953345    7037 generic.go:356] "Generic (PLEG): container finished" podID="766bc989-145d-42c9-b2d9-4916f69186c6" containerID="2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e" exitCode=0
Oct 30 21:36:52.953585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.953386    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"
Oct 30 21:36:52.953585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.953432    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" event={"ID":"766bc989-145d-42c9-b2d9-4916f69186c6","Type":"ContainerDied","Data":"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"}
Oct 30 21:36:52.953585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.953486    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55" event={"ID":"766bc989-145d-42c9-b2d9-4916f69186c6","Type":"ContainerDied","Data":"bcd68c534cbad84645f0b94be11751c6411a1adbe5a92ad0f54ab10799b05f91"}
Oct 30 21:36:52.953585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.953532    7037 scope.go:122] "RemoveContainer" containerID="2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"
Oct 30 21:36:52.955095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.955069    7037 generic.go:356] "Generic (PLEG): container finished" podID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerID="cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b" exitCode=0
Oct 30 21:36:52.955202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.955119    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"
Oct 30 21:36:52.955202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.955159    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" event={"ID":"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c","Type":"ContainerDied","Data":"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"}
Oct 30 21:36:52.955297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.955206    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg" event={"ID":"49eb04a7-c42c-4eb7-9f58-a94d1e345e2c","Type":"ContainerDied","Data":"505dbc219d235757e2898b6d5ab5def5e8cc5a6f06e8380959f863a3af90d310"}
Oct 30 21:36:52.980164 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.980139    7037 scope.go:122] "RemoveContainer" containerID="2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"
Oct 30 21:36:52.980539 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:52.980514    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e\": container with ID starting with 2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e not found: ID does not exist" containerID="2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"
Oct 30 21:36:52.980634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.980550    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e"} err="failed to get container status \"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e\": rpc error: code = NotFound desc = could not find container \"2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e\": container with ID starting with 2b87441558789a81be527e13b25d874db7d1a38c3ee8b83872815e686ac8428e not found: ID does not exist"
Oct 30 21:36:52.980634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.980577    7037 scope.go:122] "RemoveContainer" containerID="cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"
Oct 30 21:36:52.992220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.992201    7037 scope.go:122] "RemoveContainer" containerID="cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"
Oct 30 21:36:52.992572 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:52.992549    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b\": container with ID starting with cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b not found: ID does not exist" containerID="cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"
Oct 30 21:36:52.992642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:52.992580    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b"} err="failed to get container status \"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b\": rpc error: code = NotFound desc = could not find container \"cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b\": container with ID starting with cfa9ffdfcd280d0a284d0152b7dbcb194b8c0de9961c3abab4a8f195c5d73c0b not found: ID does not exist"
Oct 30 21:36:53.031394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:53.031359    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:36:53.042850 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:53.042818    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79c24a-666f6f8fbc-5fw55"]
Oct 30 21:36:53.077738 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:53.077704    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:36:53.082418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:53.082390    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79c24a-5bcf5c7bd8-gsvxg"]
Oct 30 21:36:54.600658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:54.600617    7037 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:36:54.601126 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:54.600683    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:36:54.709431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:54.709401    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" path="/var/lib/kubelet/pods/49eb04a7-c42c-4eb7-9f58-a94d1e345e2c/volumes"
Oct 30 21:36:54.709919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:54.709900    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" path="/var/lib/kubelet/pods/766bc989-145d-42c9-b2d9-4916f69186c6/volumes"
Oct 30 21:36:55.596474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.596444    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:36:55.766062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766032    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766089    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766140    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766195    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766349    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766417    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.766534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766518    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766573    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766642    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766663    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766683    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\") pod \"57593a4e-18af-4007-958b-4af655945051\" (UID: \"57593a4e-18af-4007-958b-4af655945051\") "
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766739    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:55.767007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.766925    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:55.768276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.768182    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57593a4e-18af-4007-958b-4af655945051-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.768276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.768213    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.768276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.768227    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.768276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.768239    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.768276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.768253    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.769711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.769672    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:55.770885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.770858    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:55.773744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.772117    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:55.773842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.773670    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl" pod "57593a4e-18af-4007-958b-4af655945051" (UID: "57593a4e-18af-4007-958b-4af655945051"). InnerVolumeSpecName "kube-api-access-w64wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:55.870074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.869991    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.870074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.870020    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57593a4e-18af-4007-958b-4af655945051-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.870074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.870031    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57593a4e-18af-4007-958b-4af655945051-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.870074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.870045    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w64wl\" (UniqueName: \"kubernetes.io/projected/57593a4e-18af-4007-958b-4af655945051-kube-api-access-w64wl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:36:55.996727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.996682    7037 generic.go:356] "Generic (PLEG): container finished" podID="57593a4e-18af-4007-958b-4af655945051" containerID="1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1" exitCode=0
Oct 30 21:36:55.996895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.996729    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"
Oct 30 21:36:55.996895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.996772    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" event={"ID":"57593a4e-18af-4007-958b-4af655945051","Type":"ContainerDied","Data":"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"}
Oct 30 21:36:55.996895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.996828    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c" event={"ID":"57593a4e-18af-4007-958b-4af655945051","Type":"ContainerDied","Data":"526768a17cc5e1d96618a9ad9202963ea972641378459d96f8e99210ff8d228e"}
Oct 30 21:36:55.996895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:55.996850    7037 scope.go:122] "RemoveContainer" containerID="1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"
Oct 30 21:36:56.009915 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:56.009896    7037 scope.go:122] "RemoveContainer" containerID="1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"
Oct 30 21:36:56.010186 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:36:56.010165    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1\": container with ID starting with 1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1 not found: ID does not exist" containerID="1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"
Oct 30 21:36:56.010254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:56.010197    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1"} err="failed to get container status \"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1\": rpc error: code = NotFound desc = could not find container \"1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1\": container with ID starting with 1d499d9b500665e794b69cd23f058b1ad980512f5ae3af6d79fc99b2696e4eb1 not found: ID does not exist"
Oct 30 21:36:56.065852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:56.065813    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:36:56.077824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:56.077792    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-rhx8c"]
Oct 30 21:36:56.712263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:36:56.712222    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57593a4e-18af-4007-958b-4af655945051" path="/var/lib/kubelet/pods/57593a4e-18af-4007-958b-4af655945051/volumes"
Oct 30 21:37:03.392341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.392307    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:37:03.392824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.392539    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" podUID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" containerName="maas-api" containerID="cri-o://22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8" gracePeriod=30
Oct 30 21:37:03.680644 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:03.680557    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:37:03.690142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.690114    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:37:03.788641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.788604    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\") pod \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") "
Oct 30 21:37:03.788849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.788810    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\") pod \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") "
Oct 30 21:37:03.789316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.788976    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\") pod \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") "
Oct 30 21:37:03.789316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.789076    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\" (UniqueName: \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\") pod \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\" (UID: \"d15cc2c4-7970-49ec-93b9-e514ee7b51bc\") "
Oct 30 21:37:03.789634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.789482    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca" pod "d15cc2c4-7970-49ec-93b9-e514ee7b51bc" (UID: "d15cc2c4-7970-49ec-93b9-e514ee7b51bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:03.790720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.790297    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:03.792583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.792544    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl" pod "d15cc2c4-7970-49ec-93b9-e514ee7b51bc" (UID: "d15cc2c4-7970-49ec-93b9-e514ee7b51bc"). InnerVolumeSpecName "kube-api-access-zdkdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:03.793800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.793773    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls" pod "d15cc2c4-7970-49ec-93b9-e514ee7b51bc" (UID: "d15cc2c4-7970-49ec-93b9-e514ee7b51bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:03.795292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.795270    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs" pod "d15cc2c4-7970-49ec-93b9-e514ee7b51bc" (UID: "d15cc2c4-7970-49ec-93b9-e514ee7b51bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:03.919577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.913767    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkdl\" (UniqueName: \"kubernetes.io/projected/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-kube-api-access-zdkdl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:03.919577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.913813    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:03.919577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:03.913829    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d15cc2c4-7970-49ec-93b9-e514ee7b51bc-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:04.056981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.056944    7037 generic.go:356] "Generic (PLEG): container finished" podID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" containerID="22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8" exitCode=0
Oct 30 21:37:04.057162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.056994    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"
Oct 30 21:37:04.057162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.057014    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" event={"ID":"d15cc2c4-7970-49ec-93b9-e514ee7b51bc","Type":"ContainerDied","Data":"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"}
Oct 30 21:37:04.057162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.057056    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm" event={"ID":"d15cc2c4-7970-49ec-93b9-e514ee7b51bc","Type":"ContainerDied","Data":"7cfe6268da2719c4efe4c6c4c643e9c43d083f333a2b870f57e3e3528909260e"}
Oct 30 21:37:04.057162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.057076    7037 scope.go:122] "RemoveContainer" containerID="22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"
Oct 30 21:37:04.069597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.069572    7037 scope.go:122] "RemoveContainer" containerID="22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"
Oct 30 21:37:04.069932 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:04.069903    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8\": container with ID starting with 22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8 not found: ID does not exist" containerID="22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"
Oct 30 21:37:04.070024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.069944    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8"} err="failed to get container status \"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8\": rpc error: code = NotFound desc = could not find container \"22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8\": container with ID starting with 22bd8fa9192aeec65cf6670f5fdae9ffe202ba0df7d966b0ff59abba922dd0d8 not found: ID does not exist"
Oct 30 21:37:04.113806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.113749    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:37:04.126187 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.126155    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79c24a-5d8c57b87d-sn7pm"]
Oct 30 21:37:04.709714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:04.709680    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" path="/var/lib/kubelet/pods/d15cc2c4-7970-49ec-93b9-e514ee7b51bc/volumes"
Oct 30 21:37:11.209234 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:11.209158    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:37:11.209800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:11.209424    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy" containerID="cri-o://f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15" gracePeriod=30
Oct 30 21:37:16.503057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.503026    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:37:16.631220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631187    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631249    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631307    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631335    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631474    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631532    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631549    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:16.631663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631564    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631668    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631783    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\") pod \"d9354139-0450-47ac-a4f1-92906cd69734\" (UID: \"d9354139-0450-47ac-a4f1-92906cd69734\") "
Oct 30 21:37:16.631867 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.631788    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:16.632044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632000    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:16.632044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632015    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:16.632154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632067    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:16.632532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632508    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.632532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632529    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.632718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632539    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.632718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632549    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9354139-0450-47ac-a4f1-92906cd69734-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.632718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.632557    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.633725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.633702    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:16.633940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.633920    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:16.634053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.634036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:16.634115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.634071    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth" pod "d9354139-0450-47ac-a4f1-92906cd69734" (UID: "d9354139-0450-47ac-a4f1-92906cd69734"). InnerVolumeSpecName "kube-api-access-h7hth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:16.733801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.733725    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9354139-0450-47ac-a4f1-92906cd69734-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.733801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.733751    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9354139-0450-47ac-a4f1-92906cd69734-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.733801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.733761    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7hth\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-kube-api-access-h7hth\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:16.733801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:16.733770    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9354139-0450-47ac-a4f1-92906cd69734-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:17.157504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.157451    7037 generic.go:356] "Generic (PLEG): container finished" podID="d9354139-0450-47ac-a4f1-92906cd69734" containerID="f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15" exitCode=0
Oct 30 21:37:17.157699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.157546    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"
Oct 30 21:37:17.157699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.157558    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" event={"ID":"d9354139-0450-47ac-a4f1-92906cd69734","Type":"ContainerDied","Data":"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"}
Oct 30 21:37:17.157699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.157644    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq" event={"ID":"d9354139-0450-47ac-a4f1-92906cd69734","Type":"ContainerDied","Data":"0c5fb2f8987e3872f914df0191014facca8a4a197bd980e93bb0294987a3f923"}
Oct 30 21:37:17.157699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.157694    7037 scope.go:122] "RemoveContainer" containerID="f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"
Oct 30 21:37:17.161527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.161469    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:37:17.161746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.161702    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" podUID="3cceade5-5911-42e1-8149-1c5839f2c4a0" containerName="maas-api" containerID="cri-o://e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4" gracePeriod=30
Oct 30 21:37:17.191725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.190471    7037 scope.go:122] "RemoveContainer" containerID="f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"
Oct 30 21:37:17.191725 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:17.190944    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15\": container with ID starting with f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15 not found: ID does not exist" containerID="f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"
Oct 30 21:37:17.191725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.190988    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15"} err="failed to get container status \"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15\": rpc error: code = NotFound desc = could not find container \"f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15\": container with ID starting with f7bcacfe105e9917e9dacbd8cb024b5af8baaaa290e19697908527efc385bc15 not found: ID does not exist"
Oct 30 21:37:17.238855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.238823    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:37:17.254219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.254186    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-jsspq"]
Oct 30 21:37:17.490951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.490914    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:37:17.548252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.548219    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\") pod \"3cceade5-5911-42e1-8149-1c5839f2c4a0\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") "
Oct 30 21:37:17.548812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.548333    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\" (UniqueName: \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\") pod \"3cceade5-5911-42e1-8149-1c5839f2c4a0\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") "
Oct 30 21:37:17.548812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.548447    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\") pod \"3cceade5-5911-42e1-8149-1c5839f2c4a0\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") "
Oct 30 21:37:17.548812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.548605    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\") pod \"3cceade5-5911-42e1-8149-1c5839f2c4a0\" (UID: \"3cceade5-5911-42e1-8149-1c5839f2c4a0\") "
Oct 30 21:37:17.549412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.549372    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca" pod "3cceade5-5911-42e1-8149-1c5839f2c4a0" (UID: "3cceade5-5911-42e1-8149-1c5839f2c4a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:17.552996 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.552609    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs" pod "3cceade5-5911-42e1-8149-1c5839f2c4a0" (UID: "3cceade5-5911-42e1-8149-1c5839f2c4a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:17.569769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.555622    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w" pod "3cceade5-5911-42e1-8149-1c5839f2c4a0" (UID: "3cceade5-5911-42e1-8149-1c5839f2c4a0"). InnerVolumeSpecName "kube-api-access-7rh7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:17.569769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.560672    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls" pod "3cceade5-5911-42e1-8149-1c5839f2c4a0" (UID: "3cceade5-5911-42e1-8149-1c5839f2c4a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:17.651420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.651382    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:17.651420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.651420    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rh7w\" (UniqueName: \"kubernetes.io/projected/3cceade5-5911-42e1-8149-1c5839f2c4a0-kube-api-access-7rh7w\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:17.651722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.651434    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cceade5-5911-42e1-8149-1c5839f2c4a0-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:17.651722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:17.651461    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cceade5-5911-42e1-8149-1c5839f2c4a0-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:18.167171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.167133    7037 generic.go:356] "Generic (PLEG): container finished" podID="3cceade5-5911-42e1-8149-1c5839f2c4a0" containerID="e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4" exitCode=0
Oct 30 21:37:18.167383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.167237    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"
Oct 30 21:37:18.167383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.167262    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" event={"ID":"3cceade5-5911-42e1-8149-1c5839f2c4a0","Type":"ContainerDied","Data":"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"}
Oct 30 21:37:18.167383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.167308    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb" event={"ID":"3cceade5-5911-42e1-8149-1c5839f2c4a0","Type":"ContainerDied","Data":"dad0c5b3d998167761a4651eb0d53c6fb18e04f9d9d3d9f7c8d3b00892292f52"}
Oct 30 21:37:18.167383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.167330    7037 scope.go:122] "RemoveContainer" containerID="e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"
Oct 30 21:37:18.179887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.179866    7037 scope.go:122] "RemoveContainer" containerID="e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"
Oct 30 21:37:18.195685 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:18.195646    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4\": container with ID starting with e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4 not found: ID does not exist" containerID="e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"
Oct 30 21:37:18.195843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.195695    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4"} err="failed to get container status \"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4\": rpc error: code = NotFound desc = could not find container \"e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4\": container with ID starting with e3df3218e08e62cf3e0ccfd595faf194e1aa2cb76a0ef98d532747861cbcdeb4 not found: ID does not exist"
Oct 30 21:37:18.233807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.233770    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:37:18.244814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.244785    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-290ede-5bdd575b7d-95nnb"]
Oct 30 21:37:18.714206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.714162    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cceade5-5911-42e1-8149-1c5839f2c4a0" path="/var/lib/kubelet/pods/3cceade5-5911-42e1-8149-1c5839f2c4a0/volumes"
Oct 30 21:37:18.715106 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:18.715081    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9354139-0450-47ac-a4f1-92906cd69734" path="/var/lib/kubelet/pods/d9354139-0450-47ac-a4f1-92906cd69734/volumes"
Oct 30 21:37:22.135911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:22.135876    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:37:22.167140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:22.167104    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:37:23.665576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:23.665534    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:37:27.138648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.138594    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" podUID="00697119-a608-4d04-9dca-ab40504da067" containerName="payload-processing" containerID="cri-o://81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041" gracePeriod=25
Oct 30 21:37:27.167589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.167548    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing" containerID="cri-o://6eb8ac5f6cf8d4aa6a5063d8af27d74761b7c6b6b3009ab6e5db4a30df1917b7" gracePeriod=25
Oct 30 21:37:27.438213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.438180    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:37:27.505545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.505481    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\" (UniqueName: \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\") pod \"00697119-a608-4d04-9dca-ab40504da067\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") "
Oct 30 21:37:27.505811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.505614    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\") pod \"00697119-a608-4d04-9dca-ab40504da067\" (UID: \"00697119-a608-4d04-9dca-ab40504da067\") "
Oct 30 21:37:27.506059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.506030    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume" pod "00697119-a608-4d04-9dca-ab40504da067" (UID: "00697119-a608-4d04-9dca-ab40504da067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:27.506216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.506201    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00697119-a608-4d04-9dca-ab40504da067-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.507729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.507697    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq" pod "00697119-a608-4d04-9dca-ab40504da067" (UID: "00697119-a608-4d04-9dca-ab40504da067"). InnerVolumeSpecName "kube-api-access-zvnvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:27.607381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:27.607341    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnvq\" (UniqueName: \"kubernetes.io/projected/00697119-a608-4d04-9dca-ab40504da067-kube-api-access-zvnvq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:28.258156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.258123    7037 generic.go:356] "Generic (PLEG): container finished" podID="00697119-a608-4d04-9dca-ab40504da067" containerID="81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041" exitCode=0
Oct 30 21:37:28.258664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.258167    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"
Oct 30 21:37:28.258664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.258197    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" event={"ID":"00697119-a608-4d04-9dca-ab40504da067","Type":"ContainerDied","Data":"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"}
Oct 30 21:37:28.258664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.258267    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r" event={"ID":"00697119-a608-4d04-9dca-ab40504da067","Type":"ContainerDied","Data":"71ef79a13e1e06b93db50984be7149f573d10b763ca8fc0a08669ebd345c2a35"}
Oct 30 21:37:28.258664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.258298    7037 scope.go:122] "RemoveContainer" containerID="81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"
Oct 30 21:37:28.280806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.280772    7037 scope.go:122] "RemoveContainer" containerID="81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"
Oct 30 21:37:28.281349 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:28.281250    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041\": container with ID starting with 81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041 not found: ID does not exist" containerID="81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"
Oct 30 21:37:28.281890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.281551    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041"} err="failed to get container status \"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041\": rpc error: code = NotFound desc = could not find container \"81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041\": container with ID starting with 81c9e2fd924dbfe2dea03dbad977e8a1bbe646f249b3e7660f6151c185712041 not found: ID does not exist"
Oct 30 21:37:28.319920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.319887    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:37:28.332378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.332345    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3b28bc-6bb979c6fd-2t48r"]
Oct 30 21:37:28.709663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:28.709631    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00697119-a608-4d04-9dca-ab40504da067" path="/var/lib/kubelet/pods/00697119-a608-4d04-9dca-ab40504da067/volumes"
Oct 30 21:37:29.156860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:29.156824    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:33.678541 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:33.678484    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:37:34.348364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:34.348324    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:37:34.383515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:34.383455    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:37:38.709170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:38.709117    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" podUID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" containerName="main" containerID="cri-o://cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e" gracePeriod=45
Oct 30 21:37:39.025871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.025841    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:37:39.156334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.156293    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:39.204706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.204671    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.204913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.204769    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\" (UniqueName: \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.204913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.204838    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.205050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205034    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.205174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205143    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:39.205302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205187    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.205302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205209    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\") pod \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\" (UID: \"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec\") "
Oct 30 21:37:39.205465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205436    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:39.205648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205567    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:39.205857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205841    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.205900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205865    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.205900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.205874    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.207259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.207238    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "kube-api-access-j5x6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:39.207501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.207464    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:39.207608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.207470    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm" pod "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" (UID: "2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:39.307762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.307673    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.307762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.307705    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.307762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.307716    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5x6q\" (UniqueName: \"kubernetes.io/projected/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec-kube-api-access-j5x6q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.345625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.345586    7037 generic.go:356] "Generic (PLEG): container finished" podID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" containerID="cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e" exitCode=0
Oct 30 21:37:39.345837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.345639    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"
Oct 30 21:37:39.345837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.345662    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" event={"ID":"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec","Type":"ContainerDied","Data":"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"}
Oct 30 21:37:39.345837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.345695    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6" event={"ID":"2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec","Type":"ContainerDied","Data":"8e1154ab860fd54e0b73b7a9ca4aeb472b918415c14f92454ab9883a5dab3a26"}
Oct 30 21:37:39.345837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.345711    7037 scope.go:122] "RemoveContainer" containerID="cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"
Oct 30 21:37:39.349559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.349504    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing" containerID="cri-o://4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241" gracePeriod=25
Oct 30 21:37:39.357439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.357423    7037 scope.go:122] "RemoveContainer" containerID="cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"
Oct 30 21:37:39.357735 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:39.357718    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e\": container with ID starting with cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e not found: ID does not exist" containerID="cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"
Oct 30 21:37:39.357783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.357742    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e"} err="failed to get container status \"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e\": rpc error: code = NotFound desc = could not find container \"cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e\": container with ID starting with cbb96ade7f08cb576a3283627e8ef6b4d4842514e2c073fc61001ad33b23379e not found: ID does not exist"
Oct 30 21:37:39.384952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.384915    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing" containerID="cri-o://1d79344c44bb8eb79751dd6066b0b13011fce3a82f819c80b2f78aa7beb603d6" gracePeriod=25
Oct 30 21:37:39.402593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.402562    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:37:39.405657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:39.405634    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c9477c/test-model-c9477c-kserve-75ff87dcd5-dvqb6"]
Oct 30 21:37:40.708878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:40.708840    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" path="/var/lib/kubelet/pods/2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec/volumes"
Oct 30 21:37:47.568052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.568017    7037 generic.go:356] "Generic (PLEG): container finished" podID="1983658c-79f1-41da-bb35-435d236894fa" containerID="6eb8ac5f6cf8d4aa6a5063d8af27d74761b7c6b6b3009ab6e5db4a30df1917b7" exitCode=0
Oct 30 21:37:47.568506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.568233    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" event={"ID":"1983658c-79f1-41da-bb35-435d236894fa","Type":"ContainerDied","Data":"6eb8ac5f6cf8d4aa6a5063d8af27d74761b7c6b6b3009ab6e5db4a30df1917b7"}
Oct 30 21:37:47.618241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.618206    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:37:47.657405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.657366    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\" (UniqueName: \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\") pod \"1983658c-79f1-41da-bb35-435d236894fa\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") "
Oct 30 21:37:47.657629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.657564    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\") pod \"1983658c-79f1-41da-bb35-435d236894fa\" (UID: \"1983658c-79f1-41da-bb35-435d236894fa\") "
Oct 30 21:37:47.658037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.658004    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume" pod "1983658c-79f1-41da-bb35-435d236894fa" (UID: "1983658c-79f1-41da-bb35-435d236894fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:47.658387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.658355    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1983658c-79f1-41da-bb35-435d236894fa-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:47.660231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.660197    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr" pod "1983658c-79f1-41da-bb35-435d236894fa" (UID: "1983658c-79f1-41da-bb35-435d236894fa"). InnerVolumeSpecName "kube-api-access-t5rxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:47.762907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:47.762859    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rxr\" (UniqueName: \"kubernetes.io/projected/1983658c-79f1-41da-bb35-435d236894fa-kube-api-access-t5rxr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:48.288627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.288591    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:48.288627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.288624    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:48.576975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.576938    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"
Oct 30 21:37:48.577460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.576979    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg" event={"ID":"1983658c-79f1-41da-bb35-435d236894fa","Type":"ContainerDied","Data":"c9db57c040edc72a3cf905a0d525024883bc7889f21496cde17c44a8b8b419c9"}
Oct 30 21:37:48.577460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.577049    7037 scope.go:122] "RemoveContainer" containerID="6eb8ac5f6cf8d4aa6a5063d8af27d74761b7c6b6b3009ab6e5db4a30df1917b7"
Oct 30 21:37:48.636221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.636182    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:37:48.649765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.649727    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3b28bc-5fbdf66cf9-8q4qg"]
Oct 30 21:37:48.710581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:48.710550    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1983658c-79f1-41da-bb35-435d236894fa" path="/var/lib/kubelet/pods/1983658c-79f1-41da-bb35-435d236894fa/volumes"
Oct 30 21:37:58.288972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:58.288933    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:58.289393 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:58.288937    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:37:59.676997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.676962    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:37:59.680857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.680826    7037 generic.go:356] "Generic (PLEG): container finished" podID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerID="4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241" exitCode=0
Oct 30 21:37:59.681018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.680871    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"
Oct 30 21:37:59.681018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.680910    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" event={"ID":"07f74923-7a74-40b9-862d-758eabfcd3e2","Type":"ContainerDied","Data":"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"}
Oct 30 21:37:59.681018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.680979    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc" event={"ID":"07f74923-7a74-40b9-862d-758eabfcd3e2","Type":"ContainerDied","Data":"cb2e40a0f1ee239dbdbd8fccbec1cfafd09e023d5e3b207c7b1b095feff4cdc0"}
Oct 30 21:37:59.681018 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.681013    7037 scope.go:122] "RemoveContainer" containerID="4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"
Oct 30 21:37:59.682714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.682691    7037 generic.go:356] "Generic (PLEG): container finished" podID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerID="1d79344c44bb8eb79751dd6066b0b13011fce3a82f819c80b2f78aa7beb603d6" exitCode=0
Oct 30 21:37:59.682831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.682767    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" event={"ID":"cdacc84e-96aa-4e22-bb76-d96f130a80c0","Type":"ContainerDied","Data":"1d79344c44bb8eb79751dd6066b0b13011fce3a82f819c80b2f78aa7beb603d6"}
Oct 30 21:37:59.692022 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.691998    7037 scope.go:122] "RemoveContainer" containerID="4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"
Oct 30 21:37:59.692295 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:37:59.692273    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241\": container with ID starting with 4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241 not found: ID does not exist" containerID="4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"
Oct 30 21:37:59.692376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.692306    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241"} err="failed to get container status \"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241\": rpc error: code = NotFound desc = could not find container \"4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241\": container with ID starting with 4b69d5a5047ca4489b856c7809da478f4378ab8ac043d76f3362c27aad30a241 not found: ID does not exist"
Oct 30 21:37:59.720911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.720887    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:37:59.860280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860247    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\") pod \"07f74923-7a74-40b9-862d-758eabfcd3e2\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") "
Oct 30 21:37:59.860571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860555    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\" (UniqueName: \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\") pod \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") "
Oct 30 21:37:59.860631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860618    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\") pod \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\" (UID: \"cdacc84e-96aa-4e22-bb76-d96f130a80c0\") "
Oct 30 21:37:59.860673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860629    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume" pod "07f74923-7a74-40b9-862d-758eabfcd3e2" (UID: "07f74923-7a74-40b9-862d-758eabfcd3e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:59.860673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860651    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\" (UniqueName: \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\") pod \"07f74923-7a74-40b9-862d-758eabfcd3e2\" (UID: \"07f74923-7a74-40b9-862d-758eabfcd3e2\") "
Oct 30 21:37:59.861006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.860969    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume" pod "cdacc84e-96aa-4e22-bb76-d96f130a80c0" (UID: "cdacc84e-96aa-4e22-bb76-d96f130a80c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:59.861211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.861190    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdacc84e-96aa-4e22-bb76-d96f130a80c0-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:59.861305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.861214    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74923-7a74-40b9-862d-758eabfcd3e2-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:59.863296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.863276    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt" pod "cdacc84e-96aa-4e22-bb76-d96f130a80c0" (UID: "cdacc84e-96aa-4e22-bb76-d96f130a80c0"). InnerVolumeSpecName "kube-api-access-vfvbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:59.863387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.863301    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm" pod "07f74923-7a74-40b9-862d-758eabfcd3e2" (UID: "07f74923-7a74-40b9-862d-758eabfcd3e2"). InnerVolumeSpecName "kube-api-access-dgfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:59.962285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.962197    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgfkm\" (UniqueName: \"kubernetes.io/projected/07f74923-7a74-40b9-862d-758eabfcd3e2-kube-api-access-dgfkm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:37:59.962285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:37:59.962230    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvbt\" (UniqueName: \"kubernetes.io/projected/cdacc84e-96aa-4e22-bb76-d96f130a80c0-kube-api-access-vfvbt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:00.033435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.033396    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:38:00.043892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.043856    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-79e911-6dc57c86dd-p2bjc"]
Oct 30 21:38:00.692640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.692560    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr" event={"ID":"cdacc84e-96aa-4e22-bb76-d96f130a80c0","Type":"ContainerDied","Data":"2addec4ab468166c22326cc721d12b6056dd93544b4028584fccacfa807d033b"}
Oct 30 21:38:00.693087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.692641    7037 scope.go:122] "RemoveContainer" containerID="1d79344c44bb8eb79751dd6066b0b13011fce3a82f819c80b2f78aa7beb603d6"
Oct 30 21:38:00.693087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.692568    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"
Oct 30 21:38:00.709876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.709846    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" path="/var/lib/kubelet/pods/07f74923-7a74-40b9-862d-758eabfcd3e2/volumes"
Oct 30 21:38:00.748995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.748964    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:38:00.760679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:00.760650    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-79e911-65d94b4767-glrjr"]
Oct 30 21:38:02.709800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:02.709758    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" path="/var/lib/kubelet/pods/cdacc84e-96aa-4e22-bb76-d96f130a80c0/volumes"
Oct 30 21:38:03.674537 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:03.674480    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:38:04.254528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.254478    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255237    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" containerName="main"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255251    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" containerName="main"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255267    7037 cpu_manager.go:414] "containerMap: removing container" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255272    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255285    7037 cpu_manager.go:414] "containerMap: removing container" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255292    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255303    7037 cpu_manager.go:414] "containerMap: removing container" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255308    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255315    7037 cpu_manager.go:414] "containerMap: removing container" podUID="00697119-a608-4d04-9dca-ab40504da067" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255321    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00697119-a608-4d04-9dca-ab40504da067" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255329    7037 cpu_manager.go:414] "containerMap: removing container" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255334    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255345    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255349    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255366    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255370    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255377    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3cceade5-5911-42e1-8149-1c5839f2c4a0" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255382    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cceade5-5911-42e1-8149-1c5839f2c4a0" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255389    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255392    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255417    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255422    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255431    7037 cpu_manager.go:414] "containerMap: removing container" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255435    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255561    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec4ad81-3aa2-4bc5-b6d5-0bdecbf932ec" containerName="main"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255575    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="57593a4e-18af-4007-958b-4af655945051" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255584    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdacc84e-96aa-4e22-bb76-d96f130a80c0" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255593    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f74923-7a74-40b9-862d-758eabfcd3e2" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255600    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="00697119-a608-4d04-9dca-ab40504da067" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255605    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="49eb04a7-c42c-4eb7-9f58-a94d1e345e2c" containerName="payload-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255612    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c297a8b-24b5-4179-a5ba-3b5ec90c851f" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255620    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15cc2c4-7970-49ec-93b9-e514ee7b51bc" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255627    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cceade5-5911-42e1-8149-1c5839f2c4a0" containerName="maas-api"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255633    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9354139-0450-47ac-a4f1-92906cd69734" containerName="istio-proxy"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255643    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1983658c-79f1-41da-bb35-435d236894fa" containerName="payload-pre-processing"
Oct 30 21:38:04.256945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.255649    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="766bc989-145d-42c9-b2d9-4916f69186c6" containerName="payload-pre-processing"
Oct 30 21:38:04.259353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.259333    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.275938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.275911    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:38:04.308677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.308595    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f5bef0-dockercfg-jp7vj\""
Oct 30 21:38:04.312301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.312274    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f5bef0\""
Oct 30 21:38:04.361271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.361220    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.361624 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.361335    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zczb\" (UniqueName: \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.462265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.462226    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.462523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.462340    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zczb\" (UniqueName: \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.464030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.463050    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.482723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.482695    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zczb\" (UniqueName: \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\") pod \"payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.575458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.575429    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:04.885666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:04.885632    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:38:05.732785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:05.732745    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" event={"ID":"c7d67858-28fe-4705-9b95-567c6d9acc57","Type":"ContainerStarted","Data":"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"}
Oct 30 21:38:05.733221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:05.732809    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" event={"ID":"c7d67858-28fe-4705-9b95-567c6d9acc57","Type":"ContainerStarted","Data":"b1c74c5127771dc61664c21aaed41df59870b8a264f89534e4a336a64119b33f"}
Oct 30 21:38:05.733221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:05.732834    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:05.787181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:05.787127    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" podStartSLOduration=1.787113877 podStartE2EDuration="1.787113877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:05.78471125 +0000 UTC m=+2229.196250686" watchObservedRunningTime="2026-10-30 21:38:05.787113877 +0000 UTC m=+2229.198653313"
Oct 30 21:38:06.416523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.416456    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:38:06.416728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.416674    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" podUID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" containerName="maas-api" containerID="cri-o://e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e" gracePeriod=30
Oct 30 21:38:06.725537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.725506    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:38:06.740909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.740868    7037 generic.go:356] "Generic (PLEG): container finished" podID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" containerID="e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e" exitCode=0
Oct 30 21:38:06.741379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.740912    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"
Oct 30 21:38:06.741379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.740929    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" event={"ID":"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2","Type":"ContainerDied","Data":"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"}
Oct 30 21:38:06.741379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.740998    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986" event={"ID":"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2","Type":"ContainerDied","Data":"3abb9f30e97fb6b6a5f5efa1e5a55b6cd04d3e973ef95dc8bd0d35e645311a61"}
Oct 30 21:38:06.741379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.741017    7037 scope.go:122] "RemoveContainer" containerID="e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"
Oct 30 21:38:06.751972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.751952    7037 scope.go:122] "RemoveContainer" containerID="e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"
Oct 30 21:38:06.752289 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:06.752267    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e\": container with ID starting with e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e not found: ID does not exist" containerID="e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"
Oct 30 21:38:06.752336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.752298    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e"} err="failed to get container status \"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e\": rpc error: code = NotFound desc = could not find container \"e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e\": container with ID starting with e4f19fbe07e9c6926d7136e4825573858b2c570f3b89da8f1a971048137eb76e not found: ID does not exist"
Oct 30 21:38:06.797196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.795980    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\") pod \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") "
Oct 30 21:38:06.797196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.796100    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\") pod \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") "
Oct 30 21:38:06.797196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.796180    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\" (UniqueName: \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\") pod \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") "
Oct 30 21:38:06.797196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.796405    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\") pod \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\" (UID: \"bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2\") "
Oct 30 21:38:06.798599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.798556    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca" pod "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" (UID: "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:06.798997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.798968    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs" pod "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" (UID: "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:06.800979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.800946    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls" pod "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" (UID: "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:06.801094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.801073    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4" pod "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" (UID: "bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2"). InnerVolumeSpecName "kube-api-access-5kxp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:06.899839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.899751    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.899839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.899784    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.899839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.899798    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.899839 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:06.899812    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxp4\" (UniqueName: \"kubernetes.io/projected/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2-kube-api-access-5kxp4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.088524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:07.088476    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:38:07.098201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:07.098167    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2628a-7c7996f54c-4v986"]
Oct 30 21:38:08.710795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:08.710764    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" path="/var/lib/kubelet/pods/bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2/volumes"
Oct 30 21:38:14.689438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.689393    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:38:14.690415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.690392    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" containerName="maas-api"
Oct 30 21:38:14.690415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.690415    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" containerName="maas-api"
Oct 30 21:38:14.690721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.690701    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9a0bbc-0ff0-4580-aa3a-ce01dcc160f2" containerName="maas-api"
Oct 30 21:38:14.696388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.696366    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715198    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715342    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715416    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715438    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715486    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.716160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.715684    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvff5\" (UniqueName: \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.717250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.717225    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:38:14.723818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.723669    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ad73a4\"/\"openshift-service-ca.crt\""
Oct 30 21:38:14.723971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.723896    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ad73a4\"/\"default-dockercfg-b49x2\""
Oct 30 21:38:14.724045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.724011    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ad73a4\"/\"auth-test-model-ad73a4-kserve-self-signed-certs\""
Oct 30 21:38:14.724410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.724392    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ad73a4\"/\"kube-root-ca.crt\""
Oct 30 21:38:14.819037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.818997    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvff5\" (UniqueName: \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819088    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819208    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819279    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819303    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819365    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.819903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.819882    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.820711 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.820684    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.821197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.821173    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.825078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.824673    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.825564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.825543    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:14.847029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:14.847000    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvff5\" (UniqueName: \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\") pod \"auth-test-model-ad73a4-kserve-6b448b4bb9-265w2\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") " pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:15.021993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:15.020738    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:15.337274 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:38:15.337243    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ccefd4_6668_4fd1_b51a_db4476bd2b08.slice/crio-3c08169cd9af8b3d120d96094efe096ecc6d9f73e69d3650b23209745ccdf503 WatchSource:0}: Error finding container 3c08169cd9af8b3d120d96094efe096ecc6d9f73e69d3650b23209745ccdf503: Status 404 returned error can't find the container with id 3c08169cd9af8b3d120d96094efe096ecc6d9f73e69d3650b23209745ccdf503
Oct 30 21:38:15.337405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:15.337324    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:38:15.830762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:15.830726    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" event={"ID":"47ccefd4-6668-4fd1-b51a-db4476bd2b08","Type":"ContainerStarted","Data":"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"}
Oct 30 21:38:15.830762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:15.830769    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" event={"ID":"47ccefd4-6668-4fd1-b51a-db4476bd2b08","Type":"ContainerStarted","Data":"3c08169cd9af8b3d120d96094efe096ecc6d9f73e69d3650b23209745ccdf503"}
Oct 30 21:38:15.871702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:15.871653    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" podStartSLOduration=1.871638109 podStartE2EDuration="1.871638109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.869971264 +0000 UTC m=+2239.281510704" watchObservedRunningTime="2026-10-30 21:38:15.871638109 +0000 UTC m=+2239.283177545"
Oct 30 21:38:16.021288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:16.021243    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:16.769531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:16.769476    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:38:24.238108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:24.238051    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:38:24.238108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:24.238372    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" containerID="cri-o://415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6" gracePeriod=30
Oct 30 21:38:24.250173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:24.250147    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:38:24.250353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:24.250318    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" containerID="cri-o://b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976" gracePeriod=30
Oct 30 21:38:25.021365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.021333    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:25.034715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.034688    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:25.349945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.349913    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:38:25.380975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.380939    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:38:25.630755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.629044    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:38:25.631065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.631022    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:38:25.926053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:25.925972    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:38:26.586222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:26.586184    7037 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:38:26.586653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:26.586244    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:38:29.544592 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.544562    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:38:29.575454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.575430    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:38:29.659917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.659886    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.659917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.659925    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.660287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.659964    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.660287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660091    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.660287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660153    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.660287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660192    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.660287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660261    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660316    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660352    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660395    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660429    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660461    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660552    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660621    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660695    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.661419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.660729    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.662004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.661813    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.662004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.661919    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\") pod \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\" (UID: \"c3b37376-68ce-49ba-8d8b-5d050b4cee2e\") "
Oct 30 21:38:29.662004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.661994    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\") pod \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\" (UID: \"12d869b7-969e-4b6b-b0ad-ac247aa8f871\") "
Oct 30 21:38:29.662155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662001    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.662213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662182    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.662661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662639    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.662746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662665    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.662746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662679    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.662978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.662959    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:29.663368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.663262    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:29.664544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.663398    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.664544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.663560    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.664544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.663788    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:29.664544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.663955    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.664544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.664420    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.665603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.665570    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:29.665603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.665581    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:29.666088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.665657    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "kube-api-access-hfhx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:29.666088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.665686    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.666776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.666754    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy" pod "c3b37376-68ce-49ba-8d8b-5d050b4cee2e" (UID: "c3b37376-68ce-49ba-8d8b-5d050b4cee2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.667113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.667076    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "kube-api-access-jn97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:29.667468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.667447    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:29.667980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.667962    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy" pod "12d869b7-969e-4b6b-b0ad-ac247aa8f871" (UID: "12d869b7-969e-4b6b-b0ad-ac247aa8f871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764371    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764407    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764419    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764433    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhx4\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-kube-api-access-hfhx4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764447    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764461    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764474    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn97t\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-kube-api-access-jn97t\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764500    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764514    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764525    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764538    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764549    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764560    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764571    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d869b7-969e-4b6b-b0ad-ac247aa8f871-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.764908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.764583    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b37376-68ce-49ba-8d8b-5d050b4cee2e-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:29.946117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.946085    7037 generic.go:356] "Generic (PLEG): container finished" podID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerID="415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6" exitCode=0
Oct 30 21:38:29.946337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.946136    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"
Oct 30 21:38:29.946337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.946174    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" event={"ID":"12d869b7-969e-4b6b-b0ad-ac247aa8f871","Type":"ContainerDied","Data":"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"}
Oct 30 21:38:29.946337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.946245    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh" event={"ID":"12d869b7-969e-4b6b-b0ad-ac247aa8f871","Type":"ContainerDied","Data":"89b508d15d06139e676f8bdebf50525a16b29a4071599b29512479100068aa21"}
Oct 30 21:38:29.946337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.946278    7037 scope.go:122] "RemoveContainer" containerID="415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"
Oct 30 21:38:29.947760 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.947740    7037 generic.go:356] "Generic (PLEG): container finished" podID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerID="b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976" exitCode=0
Oct 30 21:38:29.947884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.947779    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" event={"ID":"c3b37376-68ce-49ba-8d8b-5d050b4cee2e","Type":"ContainerDied","Data":"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"}
Oct 30 21:38:29.947884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.947834    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp" event={"ID":"c3b37376-68ce-49ba-8d8b-5d050b4cee2e","Type":"ContainerDied","Data":"1f70576faaf26126e04a72760498cb86f599488d39cdb227a9d784f047128719"}
Oct 30 21:38:29.947884 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.947785    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"
Oct 30 21:38:29.960161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.960145    7037 scope.go:122] "RemoveContainer" containerID="415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"
Oct 30 21:38:29.960473 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:29.960446    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6\": container with ID starting with 415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6 not found: ID does not exist" containerID="415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"
Oct 30 21:38:29.960534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.960482    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6"} err="failed to get container status \"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6\": rpc error: code = NotFound desc = could not find container \"415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6\": container with ID starting with 415162be981e8b5c9ec4d661ee439b9d0aed1f1ffd847d2d37ec5901ee6d90d6 not found: ID does not exist"
Oct 30 21:38:29.960534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.960525    7037 scope.go:122] "RemoveContainer" containerID="b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"
Oct 30 21:38:29.985171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.985149    7037 scope.go:122] "RemoveContainer" containerID="b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"
Oct 30 21:38:29.985484 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:29.985465    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976\": container with ID starting with b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976 not found: ID does not exist" containerID="b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"
Oct 30 21:38:29.985484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:29.985506    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976"} err="failed to get container status \"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976\": rpc error: code = NotFound desc = could not find container \"b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976\": container with ID starting with b4c053d547e6f56736ea78acc1b774bbcb892569d9df1c791340ffcc0e88d976 not found: ID does not exist"
Oct 30 21:38:30.008595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.008560    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:38:30.020326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.020247    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-gknmh"]
Oct 30 21:38:30.048607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.048570    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:38:30.052892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.052864    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-6zbxp"]
Oct 30 21:38:30.350314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.350269    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" podUID="884587fa-e1c5-4747-bf30-33f4392b6c6f" containerName="payload-processing" containerID="cri-o://931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab" gracePeriod=25
Oct 30 21:38:30.381942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.381905    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing" containerID="cri-o://fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf" gracePeriod=25
Oct 30 21:38:30.662258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.662231    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:38:30.711228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.711197    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" path="/var/lib/kubelet/pods/12d869b7-969e-4b6b-b0ad-ac247aa8f871/volumes"
Oct 30 21:38:30.712156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.712130    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" path="/var/lib/kubelet/pods/c3b37376-68ce-49ba-8d8b-5d050b4cee2e/volumes"
Oct 30 21:38:30.761401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.761373    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:38:30.763941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763904    7037 cpu_manager.go:414] "containerMap: removing container" podUID="884587fa-e1c5-4747-bf30-33f4392b6c6f" containerName="payload-processing"
Oct 30 21:38:30.763941 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763933    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884587fa-e1c5-4747-bf30-33f4392b6c6f" containerName="payload-processing"
Oct 30 21:38:30.764173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763954    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy"
Oct 30 21:38:30.764173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763963    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy"
Oct 30 21:38:30.764173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763981    7037 cpu_manager.go:414] "containerMap: removing container" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy"
Oct 30 21:38:30.764173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.763989    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy"
Oct 30 21:38:30.764421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.764200    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="884587fa-e1c5-4747-bf30-33f4392b6c6f" containerName="payload-processing"
Oct 30 21:38:30.764421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.764215    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b37376-68ce-49ba-8d8b-5d050b4cee2e" containerName="istio-proxy"
Oct 30 21:38:30.764421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.764227    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d869b7-969e-4b6b-b0ad-ac247aa8f871" containerName="istio-proxy"
Oct 30 21:38:30.802300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.794837    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\" (UniqueName: \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\") pod \"884587fa-e1c5-4747-bf30-33f4392b6c6f\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") "
Oct 30 21:38:30.802300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.795423    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\") pod \"884587fa-e1c5-4747-bf30-33f4392b6c6f\" (UID: \"884587fa-e1c5-4747-bf30-33f4392b6c6f\") "
Oct 30 21:38:30.802300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.796462    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume" pod "884587fa-e1c5-4747-bf30-33f4392b6c6f" (UID: "884587fa-e1c5-4747-bf30-33f4392b6c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:30.829770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.829206    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8" pod "884587fa-e1c5-4747-bf30-33f4392b6c6f" (UID: "884587fa-e1c5-4747-bf30-33f4392b6c6f"). InnerVolumeSpecName "kube-api-access-z9kf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:30.838214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.837394    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:38:30.838214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.837564    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:30.854857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.854576    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-66cc54-dockercfg-7589p\""
Oct 30 21:38:30.854857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.854683    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-66cc54\""
Oct 30 21:38:30.897046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.896980    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9kf8\" (UniqueName: \"kubernetes.io/projected/884587fa-e1c5-4747-bf30-33f4392b6c6f-kube-api-access-z9kf8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:30.897046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.897006    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884587fa-e1c5-4747-bf30-33f4392b6c6f-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:30.957090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.957058    7037 generic.go:356] "Generic (PLEG): container finished" podID="884587fa-e1c5-4747-bf30-33f4392b6c6f" containerID="931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab" exitCode=0
Oct 30 21:38:30.957243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.957100    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"
Oct 30 21:38:30.957243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.957133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" event={"ID":"884587fa-e1c5-4747-bf30-33f4392b6c6f","Type":"ContainerDied","Data":"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"}
Oct 30 21:38:30.957243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.957197    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4" event={"ID":"884587fa-e1c5-4747-bf30-33f4392b6c6f","Type":"ContainerDied","Data":"9d27ba4d77ff1fa9d293c5445e2217a876178e94c7f9ad14fda8a8d6ecb68666"}
Oct 30 21:38:30.957243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.957222    7037 scope.go:122] "RemoveContainer" containerID="931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"
Oct 30 21:38:30.969019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.969003    7037 scope.go:122] "RemoveContainer" containerID="931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"
Oct 30 21:38:30.969304 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:30.969281    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab\": container with ID starting with 931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab not found: ID does not exist" containerID="931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"
Oct 30 21:38:30.969390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.969312    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab"} err="failed to get container status \"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab\": rpc error: code = NotFound desc = could not find container \"931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab\": container with ID starting with 931235f1ecffc1cf740f24d3294769d2277232509e839511c338e2f6179af9ab not found: ID does not exist"
Oct 30 21:38:30.998732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.998706    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:30.998866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:30.998741    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rwv\" (UniqueName: \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.010564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.010537    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:38:31.014689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.014668    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-abf04b-c4db67488-6jzt4"]
Oct 30 21:38:31.070901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.070860    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:38:31.100363 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.100329    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.100553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.100368    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rwv\" (UniqueName: \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.101111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.101093    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.119934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.119908    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rwv\" (UniqueName: \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\") pod \"payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.175233 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.175150    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:31.483096 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:38:31.483062    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ddf27d_ae2e_4c81_9153_060f55931652.slice/crio-ae32fea308397b1866f90a22b8c0fdd6e435c58f5583193fd11c1fe6558dcbea WatchSource:0}: Error finding container ae32fea308397b1866f90a22b8c0fdd6e435c58f5583193fd11c1fe6558dcbea: Status 404 returned error can't find the container with id ae32fea308397b1866f90a22b8c0fdd6e435c58f5583193fd11c1fe6558dcbea
Oct 30 21:38:31.483881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.483859    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:38:31.968869 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.968835    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" event={"ID":"68ddf27d-ae2e-4c81-9153-060f55931652","Type":"ContainerStarted","Data":"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"}
Oct 30 21:38:31.969329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.968892    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" event={"ID":"68ddf27d-ae2e-4c81-9153-060f55931652","Type":"ContainerStarted","Data":"ae32fea308397b1866f90a22b8c0fdd6e435c58f5583193fd11c1fe6558dcbea"}
Oct 30 21:38:31.969329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:31.968955    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:32.011667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:32.011624    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" podStartSLOduration=2.011611093 podStartE2EDuration="2.011611093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:32.008632798 +0000 UTC m=+2255.420172235" watchObservedRunningTime="2026-10-30 21:38:32.011611093 +0000 UTC m=+2255.423150529"
Oct 30 21:38:32.708154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:32.708119    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884587fa-e1c5-4747-bf30-33f4392b6c6f" path="/var/lib/kubelet/pods/884587fa-e1c5-4747-bf30-33f4392b6c6f/volumes"
Oct 30 21:38:33.193601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.193561    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:38:33.198323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.198305    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.223418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.223388    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-f5bef0-serving-cert\""
Oct 30 21:38:33.228935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.228905    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:38:33.331134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.331100    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqtr\" (UniqueName: \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.331312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.331235    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.331359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.331347    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.331469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.331447    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.433041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.433008    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.433226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.433114    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.433226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.433155    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.433343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.433266    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqtr\" (UniqueName: \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.433843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.433817    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.435569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.435546    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.435639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.435626    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.453003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.452948    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqtr\" (UniqueName: \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\") pod \"maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.513797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.513762    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:33.673551 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:33.673511    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:38:33.817748 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:38:33.817715    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967aa72f_f0ad_4f40_8559_37a251c093c3.slice/crio-c17b004333123f08885672752d84bb7bd225292d4f36f8b8f4fa44098f52ceef WatchSource:0}: Error finding container c17b004333123f08885672752d84bb7bd225292d4f36f8b8f4fa44098f52ceef: Status 404 returned error can't find the container with id c17b004333123f08885672752d84bb7bd225292d4f36f8b8f4fa44098f52ceef
Oct 30 21:38:33.819481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.819455    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:38:33.819826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.819794    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:38:33.996908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:33.996873    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" event={"ID":"967aa72f-f0ad-4f40-8559-37a251c093c3","Type":"ContainerStarted","Data":"c17b004333123f08885672752d84bb7bd225292d4f36f8b8f4fa44098f52ceef"}
Oct 30 21:38:35.006499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:35.006456    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" event={"ID":"967aa72f-f0ad-4f40-8559-37a251c093c3","Type":"ContainerStarted","Data":"a45498134c55275d7e1557a19fbd7c46803785e4431654fcee37a4ffad71911e"}
Oct 30 21:38:35.006987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:35.006559    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:35.056122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:35.056073    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" podStartSLOduration=1.667717438 podStartE2EDuration="2.056058857s" totalImagesPullingTime="388.341419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:35.055815347 +0000 UTC m=+2258.467354772" watchObservedRunningTime="2026-10-30 21:38:35.056058857 +0000 UTC m=+2258.467598291"
Oct 30 21:38:41.020347 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:41.020257    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:38:41.070632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:41.070589    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:38:42.522377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.522341    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:38:42.522899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.522692    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" podUID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" containerName="maas-api" containerID="cri-o://949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f" gracePeriod=30
Oct 30 21:38:42.855029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.854991    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:38:42.985979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.985944    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\" (UniqueName: \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\") pod \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") "
Oct 30 21:38:42.986177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.986156    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\") pod \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") "
Oct 30 21:38:42.986286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.986213    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\") pod \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") "
Oct 30 21:38:42.986349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.986327    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\") pod \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\" (UID: \"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04\") "
Oct 30 21:38:42.987417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.987390    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca" pod "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" (UID: "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:42.990640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.990578    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls" pod "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" (UID: "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:42.993374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.993343    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs" pod "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" (UID: "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:42.993806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.993783    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:38:42.995925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:42.995893    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s" pod "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" (UID: "6a0c4efa-19b2-44a5-8891-6a55dc1bfb04"). InnerVolumeSpecName "kube-api-access-9dh5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:43.076367 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.076326    7037 generic.go:356] "Generic (PLEG): container finished" podID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" containerID="949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f" exitCode=0
Oct 30 21:38:43.076583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.076451    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" event={"ID":"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04","Type":"ContainerDied","Data":"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"}
Oct 30 21:38:43.076583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.076546    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np" event={"ID":"6a0c4efa-19b2-44a5-8891-6a55dc1bfb04","Type":"ContainerDied","Data":"14b048ea0f2ecd1177c8d8d92ee277e4593737dfa4a800ca7128b6e566dd6456"}
Oct 30 21:38:43.076583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.076580    7037 scope.go:122] "RemoveContainer" containerID="949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"
Oct 30 21:38:43.076786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.076750    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"
Oct 30 21:38:43.088545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.088448    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:43.088545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.088478    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:43.088545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.088505    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:43.088545 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.088520    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dh5s\" (UniqueName: \"kubernetes.io/projected/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04-kube-api-access-9dh5s\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:43.098160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.098130    7037 scope.go:122] "RemoveContainer" containerID="949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"
Oct 30 21:38:43.098455 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:43.098432    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f\": container with ID starting with 949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f not found: ID does not exist" containerID="949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"
Oct 30 21:38:43.098578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.098466    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f"} err="failed to get container status \"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f\": rpc error: code = NotFound desc = could not find container \"949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f\": container with ID starting with 949cf2c47b26d5c95f1aabfd225b94872c85155b82bff64f5456a8d77643c73f not found: ID does not exist"
Oct 30 21:38:43.137070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.137038    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:38:43.143783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:43.143758    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-abf04b-bdd5bdd84-856np"]
Oct 30 21:38:44.710183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:44.710140    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" path="/var/lib/kubelet/pods/6a0c4efa-19b2-44a5-8891-6a55dc1bfb04/volumes"
Oct 30 21:38:50.670441 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.670414    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:38:50.795146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.795107    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\") pod \"cc6f6946-ba4c-4800-a521-911fae1f33e1\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") "
Oct 30 21:38:50.795347 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.795292    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\" (UniqueName: \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\") pod \"cc6f6946-ba4c-4800-a521-911fae1f33e1\" (UID: \"cc6f6946-ba4c-4800-a521-911fae1f33e1\") "
Oct 30 21:38:50.796117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.795581    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume" pod "cc6f6946-ba4c-4800-a521-911fae1f33e1" (UID: "cc6f6946-ba4c-4800-a521-911fae1f33e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:50.799992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.798158    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f6946-ba4c-4800-a521-911fae1f33e1-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:50.831578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.831432    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7" pod "cc6f6946-ba4c-4800-a521-911fae1f33e1" (UID: "cc6f6946-ba4c-4800-a521-911fae1f33e1"). InnerVolumeSpecName "kube-api-access-f8nj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:50.900116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:50.900025    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nj7\" (UniqueName: \"kubernetes.io/projected/cc6f6946-ba4c-4800-a521-911fae1f33e1-kube-api-access-f8nj7\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.153999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.153916    7037 generic.go:356] "Generic (PLEG): container finished" podID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerID="fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf" exitCode=0
Oct 30 21:38:51.153999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.153986    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"
Oct 30 21:38:51.154251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.153998    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" event={"ID":"cc6f6946-ba4c-4800-a521-911fae1f33e1","Type":"ContainerDied","Data":"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"}
Oct 30 21:38:51.154251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.154063    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb" event={"ID":"cc6f6946-ba4c-4800-a521-911fae1f33e1","Type":"ContainerDied","Data":"01feab42614afe7fd97d2bbedf2146e0d68305e1d0373dd066a8cb05d471da77"}
Oct 30 21:38:51.154251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.154094    7037 scope.go:122] "RemoveContainer" containerID="fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"
Oct 30 21:38:51.167298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.167278    7037 scope.go:122] "RemoveContainer" containerID="fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"
Oct 30 21:38:51.167604 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:38:51.167582    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf\": container with ID starting with fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf not found: ID does not exist" containerID="fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"
Oct 30 21:38:51.167691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.167613    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf"} err="failed to get container status \"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf\": rpc error: code = NotFound desc = could not find container \"fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf\": container with ID starting with fee79ece0f758a9cccc687a062adaa6a3ee15ace4ca99901d786b917a576cbcf not found: ID does not exist"
Oct 30 21:38:51.209774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.209730    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:38:51.225780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:51.223478    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-abf04b-566d865847-m6kgb"]
Oct 30 21:38:52.708995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:52.708962    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" path="/var/lib/kubelet/pods/cc6f6946-ba4c-4800-a521-911fae1f33e1/volumes"
Oct 30 21:38:56.711585 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:38:56.711556    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:39:01.711858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:01.711812    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing" containerID="cri-o://1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170" gracePeriod=25
Oct 30 21:39:03.672299 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:03.672258    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:39:05.116971 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:05.116936    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:39:15.063905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.063865    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:39:15.064634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064615    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" containerName="maas-api"
Oct 30 21:39:15.064634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064633    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" containerName="maas-api"
Oct 30 21:39:15.064805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064684    7037 cpu_manager.go:414] "containerMap: removing container" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing"
Oct 30 21:39:15.064805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064691    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing"
Oct 30 21:39:15.064805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064800    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6f6946-ba4c-4800-a521-911fae1f33e1" containerName="payload-pre-processing"
Oct 30 21:39:15.064970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.064812    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0c4efa-19b2-44a5-8891-6a55dc1bfb04" containerName="maas-api"
Oct 30 21:39:15.070224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.070199    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.096386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.096360    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:39:15.118194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.118158    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:39:15.156701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.156666    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-889195\""
Oct 30 21:39:15.157113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.157091    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-889195-dockercfg-52cw9\""
Oct 30 21:39:15.257987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.257955    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.258169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.258112    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvtr\" (UniqueName: \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.359790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.359756    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.359961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.359894    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvtr\" (UniqueName: \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.360435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.360413    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.379623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.379596    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvtr\" (UniqueName: \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\") pod \"payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.384638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.384621    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:15.683797 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:39:15.683765    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf72b143_ed3f_4652_9c1d_cf400e8458a9.slice/crio-1eb8cf7045a03c5992f5c63cce2c065a3f588c2592e01826dcf490cc1ad9ffb5 WatchSource:0}: Error finding container 1eb8cf7045a03c5992f5c63cce2c065a3f588c2592e01826dcf490cc1ad9ffb5: Status 404 returned error can't find the container with id 1eb8cf7045a03c5992f5c63cce2c065a3f588c2592e01826dcf490cc1ad9ffb5
Oct 30 21:39:15.684692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:15.684669    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:39:16.340603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.340560    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" event={"ID":"df72b143-ed3f-4652-9c1d-cf400e8458a9","Type":"ContainerStarted","Data":"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"}
Oct 30 21:39:16.341079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.340738    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:16.341079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.340776    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" event={"ID":"df72b143-ed3f-4652-9c1d-cf400e8458a9","Type":"ContainerStarted","Data":"1eb8cf7045a03c5992f5c63cce2c065a3f588c2592e01826dcf490cc1ad9ffb5"}
Oct 30 21:39:16.384687 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.384632    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" podStartSLOduration=1.384619536 podStartE2EDuration="1.384619536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.381330772 +0000 UTC m=+2299.792870208" watchObservedRunningTime="2026-10-30 21:39:16.384619536 +0000 UTC m=+2299.796158973"
Oct 30 21:39:16.753892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.753818    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:39:16.754041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:16.754000    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" podUID="61b7b05f-29eb-4184-b612-7fef28999342" containerName="maas-api" containerID="cri-o://3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2" gracePeriod=30
Oct 30 21:39:17.074024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.073999    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:39:17.104341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.104302    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\") pod \"61b7b05f-29eb-4184-b612-7fef28999342\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") "
Oct 30 21:39:17.104539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.104388    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\") pod \"61b7b05f-29eb-4184-b612-7fef28999342\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") "
Oct 30 21:39:17.104539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.104423    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\") pod \"61b7b05f-29eb-4184-b612-7fef28999342\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") "
Oct 30 21:39:17.104683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.104582    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\" (UniqueName: \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\") pod \"61b7b05f-29eb-4184-b612-7fef28999342\" (UID: \"61b7b05f-29eb-4184-b612-7fef28999342\") "
Oct 30 21:39:17.104805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.104782    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca" pod "61b7b05f-29eb-4184-b612-7fef28999342" (UID: "61b7b05f-29eb-4184-b612-7fef28999342"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:17.105203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.105181    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7b05f-29eb-4184-b612-7fef28999342-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:17.106831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.106800    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl" pod "61b7b05f-29eb-4184-b612-7fef28999342" (UID: "61b7b05f-29eb-4184-b612-7fef28999342"). InnerVolumeSpecName "kube-api-access-lcvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:17.106831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.106805    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs" pod "61b7b05f-29eb-4184-b612-7fef28999342" (UID: "61b7b05f-29eb-4184-b612-7fef28999342"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:17.107000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.106979    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls" pod "61b7b05f-29eb-4184-b612-7fef28999342" (UID: "61b7b05f-29eb-4184-b612-7fef28999342"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:17.206900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.206867    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvrl\" (UniqueName: \"kubernetes.io/projected/61b7b05f-29eb-4184-b612-7fef28999342-kube-api-access-lcvrl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:17.206900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.206898    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:17.206900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.206909    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b7b05f-29eb-4184-b612-7fef28999342-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:17.349014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.348982    7037 generic.go:356] "Generic (PLEG): container finished" podID="61b7b05f-29eb-4184-b612-7fef28999342" containerID="3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2" exitCode=0
Oct 30 21:39:17.349469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.349027    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"
Oct 30 21:39:17.349469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.349077    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" event={"ID":"61b7b05f-29eb-4184-b612-7fef28999342","Type":"ContainerDied","Data":"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"}
Oct 30 21:39:17.349469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.349128    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j" event={"ID":"61b7b05f-29eb-4184-b612-7fef28999342","Type":"ContainerDied","Data":"57f55826ccccb02c7689152650294e1b17e6e035e0e0023940144383ab9cfc5f"}
Oct 30 21:39:17.349469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.349150    7037 scope.go:122] "RemoveContainer" containerID="3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"
Oct 30 21:39:17.364481 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.363695    7037 scope.go:122] "RemoveContainer" containerID="3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"
Oct 30 21:39:17.367783 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:17.367751    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2\": container with ID starting with 3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2 not found: ID does not exist" containerID="3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"
Oct 30 21:39:17.367913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.367795    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2"} err="failed to get container status \"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2\": rpc error: code = NotFound desc = could not find container \"3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2\": container with ID starting with 3e37ad67fd63acc2fbdf64ab2acdd6ecb4887cd5e7c1684227b477e9222c3fc2 not found: ID does not exist"
Oct 30 21:39:17.405932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.405898    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:39:17.413657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:17.413630    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-67f8e0-cf9cc6956-gtl5j"]
Oct 30 21:39:18.709783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:18.709754    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b7b05f-29eb-4184-b612-7fef28999342" path="/var/lib/kubelet/pods/61b7b05f-29eb-4184-b612-7fef28999342/volumes"
Oct 30 21:39:20.239655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.239618    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:39:20.240408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.240389    7037 cpu_manager.go:414] "containerMap: removing container" podUID="61b7b05f-29eb-4184-b612-7fef28999342" containerName="maas-api"
Oct 30 21:39:20.240408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.240406    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b7b05f-29eb-4184-b612-7fef28999342" containerName="maas-api"
Oct 30 21:39:20.240634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.240554    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b7b05f-29eb-4184-b612-7fef28999342" containerName="maas-api"
Oct 30 21:39:20.244207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.244183    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.279143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.279115    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e11bef\"/\"openshift-service-ca.crt\""
Oct 30 21:39:20.279299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.279115    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e11bef\"/\"default-dockercfg-qksmh\""
Oct 30 21:39:20.279299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.279199    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e11bef\"/\"auth-test-model-e11bef-kserve-self-signed-certs\""
Oct 30 21:39:20.279299 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.279230    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e11bef\"/\"kube-root-ca.crt\""
Oct 30 21:39:20.284327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.284297    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:39:20.349096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349060    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.349096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349099    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.349995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349130    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.349995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349214    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwszv\" (UniqueName: \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.349995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349329    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.349995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.349440    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450507    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450548    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450580    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450599    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwszv\" (UniqueName: \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450642    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.450823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.450719    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.451056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.451021    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.451118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.451070    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.451248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.451231    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.453071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.453047    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.453190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.453143    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.471570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.471544    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwszv\" (UniqueName: \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\") pod \"auth-test-model-e11bef-kserve-57b9dbff89-mp4sx\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") " pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.558852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.558751    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:20.861180 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:39:20.861146    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb9f8833_dc89_4c8d_84ba_d613cebadc1c.slice/crio-8eae83d4c98bb616482df42512c326b6e85315e83bbe9c4362af1632fbad91ae WatchSource:0}: Error finding container 8eae83d4c98bb616482df42512c326b6e85315e83bbe9c4362af1632fbad91ae: Status 404 returned error can't find the container with id 8eae83d4c98bb616482df42512c326b6e85315e83bbe9c4362af1632fbad91ae
Oct 30 21:39:20.861831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:20.861806    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:39:21.379195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:21.379157    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" event={"ID":"cb9f8833-dc89-4c8d-84ba-d613cebadc1c","Type":"ContainerStarted","Data":"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"}
Oct 30 21:39:21.379729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:21.379207    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" event={"ID":"cb9f8833-dc89-4c8d-84ba-d613cebadc1c","Type":"ContainerStarted","Data":"8eae83d4c98bb616482df42512c326b6e85315e83bbe9c4362af1632fbad91ae"}
Oct 30 21:39:21.419933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:21.419881    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" podStartSLOduration=1.419867163 podStartE2EDuration="1.419867163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:21.418745869 +0000 UTC m=+2304.830285367" watchObservedRunningTime="2026-10-30 21:39:21.419867163 +0000 UTC m=+2304.831406599"
Oct 30 21:39:21.559443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:21.559411    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:22.004577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.004554    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:39:22.075323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.075286    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\" (UniqueName: \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\") pod \"ad902be3-5438-4eec-9944-2038a4f1ce19\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") "
Oct 30 21:39:22.075526 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.075508    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\") pod \"ad902be3-5438-4eec-9944-2038a4f1ce19\" (UID: \"ad902be3-5438-4eec-9944-2038a4f1ce19\") "
Oct 30 21:39:22.075893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.075869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume" pod "ad902be3-5438-4eec-9944-2038a4f1ce19" (UID: "ad902be3-5438-4eec-9944-2038a4f1ce19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:22.076130 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.076112    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad902be3-5438-4eec-9944-2038a4f1ce19-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:22.077378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.077352    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt" pod "ad902be3-5438-4eec-9944-2038a4f1ce19" (UID: "ad902be3-5438-4eec-9944-2038a4f1ce19"). InnerVolumeSpecName "kube-api-access-pwmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:22.177610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.177575    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwmrt\" (UniqueName: \"kubernetes.io/projected/ad902be3-5438-4eec-9944-2038a4f1ce19-kube-api-access-pwmrt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:22.387466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.387432    7037 generic.go:356] "Generic (PLEG): container finished" podID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerID="1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170" exitCode=0
Oct 30 21:39:22.387989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.387486    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"
Oct 30 21:39:22.387989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.387530    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" event={"ID":"ad902be3-5438-4eec-9944-2038a4f1ce19","Type":"ContainerDied","Data":"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"}
Oct 30 21:39:22.387989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.387583    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q" event={"ID":"ad902be3-5438-4eec-9944-2038a4f1ce19","Type":"ContainerDied","Data":"6442bd4c01e6c6c3e138b923036cc58587c7215b1de63292d09b16899083b602"}
Oct 30 21:39:22.387989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.387604    7037 scope.go:122] "RemoveContainer" containerID="1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"
Oct 30 21:39:22.399903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.399881    7037 scope.go:122] "RemoveContainer" containerID="1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"
Oct 30 21:39:22.400170 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:22.400150    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170\": container with ID starting with 1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170 not found: ID does not exist" containerID="1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"
Oct 30 21:39:22.400236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.400185    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170"} err="failed to get container status \"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170\": rpc error: code = NotFound desc = could not find container \"1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170\": container with ID starting with 1de148ddb710284774432c732b4ea1ef33ad35890e8b60f94c9b02c60d3a5170 not found: ID does not exist"
Oct 30 21:39:22.444667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.443688    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:39:22.454805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.454773    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67f8e0-55b7f4dc67-2jd6q"]
Oct 30 21:39:22.709637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:22.709554    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" path="/var/lib/kubelet/pods/ad902be3-5438-4eec-9944-2038a4f1ce19/volumes"
Oct 30 21:39:27.352641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:27.352609    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:39:30.559823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:30.559788    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:30.572586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:30.572561    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:31.465390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:31.465353    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:39:33.673833 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:33.673793    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:39:35.822541 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:35.822504    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:39:35.856866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:35.856830    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:39:36.314031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:36.313953    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:39:40.825708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:40.825662    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" podUID="fc15ba92-ed3a-4f5c-9335-029de83660c6" containerName="payload-processing" containerID="cri-o://d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0" gracePeriod=25
Oct 30 21:39:40.858101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:40.858064    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing" containerID="cri-o://f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc" gracePeriod=25
Oct 30 21:39:41.136673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.136647    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:39:41.245077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.245042    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\") pod \"fc15ba92-ed3a-4f5c-9335-029de83660c6\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") "
Oct 30 21:39:41.245279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.245097    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\" (UniqueName: \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\") pod \"fc15ba92-ed3a-4f5c-9335-029de83660c6\" (UID: \"fc15ba92-ed3a-4f5c-9335-029de83660c6\") "
Oct 30 21:39:41.246161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.246134    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume" pod "fc15ba92-ed3a-4f5c-9335-029de83660c6" (UID: "fc15ba92-ed3a-4f5c-9335-029de83660c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:41.249612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.249572    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch" pod "fc15ba92-ed3a-4f5c-9335-029de83660c6" (UID: "fc15ba92-ed3a-4f5c-9335-029de83660c6"). InnerVolumeSpecName "kube-api-access-t9rch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:41.347052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.347021    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc15ba92-ed3a-4f5c-9335-029de83660c6-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:41.347052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.347051    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rch\" (UniqueName: \"kubernetes.io/projected/fc15ba92-ed3a-4f5c-9335-029de83660c6-kube-api-access-t9rch\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:41.585274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.585244    7037 generic.go:356] "Generic (PLEG): container finished" podID="fc15ba92-ed3a-4f5c-9335-029de83660c6" containerID="d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0" exitCode=0
Oct 30 21:39:41.585450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.585309    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"
Oct 30 21:39:41.585450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.585335    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" event={"ID":"fc15ba92-ed3a-4f5c-9335-029de83660c6","Type":"ContainerDied","Data":"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"}
Oct 30 21:39:41.585450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.585402    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq" event={"ID":"fc15ba92-ed3a-4f5c-9335-029de83660c6","Type":"ContainerDied","Data":"df8600b09419f1c9ab828fbf4c95295b25af496181798eaaf6ebd4d73157236e"}
Oct 30 21:39:41.585450 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.585432    7037 scope.go:122] "RemoveContainer" containerID="d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"
Oct 30 21:39:41.598556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.598537    7037 scope.go:122] "RemoveContainer" containerID="d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"
Oct 30 21:39:41.598861 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:41.598830    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0\": container with ID starting with d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0 not found: ID does not exist" containerID="d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"
Oct 30 21:39:41.598930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.598872    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0"} err="failed to get container status \"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0\": rpc error: code = NotFound desc = could not find container \"d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0\": container with ID starting with d6b8a87bf2d1962f087f9e66512acc58db925d33237349101b6a6eecacff2ee0 not found: ID does not exist"
Oct 30 21:39:41.637431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.637400    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:39:41.649696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:41.649672    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f23eae-86bbf64bdb-cxtqq"]
Oct 30 21:39:42.148981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.148944    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:39:42.149710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149694    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fc15ba92-ed3a-4f5c-9335-029de83660c6" containerName="payload-processing"
Oct 30 21:39:42.149710 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149711    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc15ba92-ed3a-4f5c-9335-029de83660c6" containerName="payload-processing"
Oct 30 21:39:42.149824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149739    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing"
Oct 30 21:39:42.149824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149745    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing"
Oct 30 21:39:42.149893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149863    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad902be3-5438-4eec-9944-2038a4f1ce19" containerName="payload-pre-processing"
Oct 30 21:39:42.149893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.149879    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc15ba92-ed3a-4f5c-9335-029de83660c6" containerName="payload-processing"
Oct 30 21:39:42.155042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.155018    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.173547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.173521    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:39:42.176232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.176210    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9cd83f\""
Oct 30 21:39:42.259764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.259724    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.259955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.259805    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbkj\" (UniqueName: \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.361398 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.361345    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.361627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.361438    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbkj\" (UniqueName: \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.362355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.362321    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.381035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.381002    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbkj\" (UniqueName: \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\") pod \"payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.470190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.470110    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:42.554880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.554837    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:39:42.708237 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.708204    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc15ba92-ed3a-4f5c-9335-029de83660c6" path="/var/lib/kubelet/pods/fc15ba92-ed3a-4f5c-9335-029de83660c6/volumes"
Oct 30 21:39:42.769682 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:39:42.769651    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318c1981_a3b1_4528_9229_c46921459fb4.slice/crio-a86138890278f3d33995ba304810fdca1541aca3a0acb36b5ed5bf33c8ea4cb8 WatchSource:0}: Error finding container a86138890278f3d33995ba304810fdca1541aca3a0acb36b5ed5bf33c8ea4cb8: Status 404 returned error can't find the container with id a86138890278f3d33995ba304810fdca1541aca3a0acb36b5ed5bf33c8ea4cb8
Oct 30 21:39:42.770784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:42.770755    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:39:43.601865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:43.601829    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" event={"ID":"318c1981-a3b1-4528-9229-c46921459fb4","Type":"ContainerStarted","Data":"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"}
Oct 30 21:39:43.602287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:43.601888    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" event={"ID":"318c1981-a3b1-4528-9229-c46921459fb4","Type":"ContainerStarted","Data":"a86138890278f3d33995ba304810fdca1541aca3a0acb36b5ed5bf33c8ea4cb8"}
Oct 30 21:39:43.602287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:43.601917    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:43.644287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:43.644237    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" podStartSLOduration=1.6442145670000001 podStartE2EDuration="1.644214567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:43.641772641 +0000 UTC m=+2327.053312079" watchObservedRunningTime="2026-10-30 21:39:43.644214567 +0000 UTC m=+2327.055754002"
Oct 30 21:39:44.641559 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.641525    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:39:44.646168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.646146    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.670088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.670054    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:39:44.677461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.677429    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-889195-serving-cert\""
Oct 30 21:39:44.793197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.793163    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.793381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.793238    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9b9t\" (UniqueName: \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.793381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.793310    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.793472 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.793399    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.895177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.895099    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.895177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.895152    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9b9t\" (UniqueName: \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.895399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.895287    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.895399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.895344    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.895980 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.895955    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.897598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.897576    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.897683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.897614    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.914698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.914673    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9b9t\" (UniqueName: \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\") pod \"maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:44.963232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:44.963201    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:45.272308 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:39:45.272279    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d09b60_e573_40c6_9d22_bdd339728f14.slice/crio-27a274b582b2e7d7eda019a87ad3db90700fbc2f76b04024ab464d6559aa7433 WatchSource:0}: Error finding container 27a274b582b2e7d7eda019a87ad3db90700fbc2f76b04024ab464d6559aa7433: Status 404 returned error can't find the container with id 27a274b582b2e7d7eda019a87ad3db90700fbc2f76b04024ab464d6559aa7433
Oct 30 21:39:45.272880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:45.272793    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:39:45.619288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:45.619261    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" event={"ID":"02d09b60-e573-40c6-9d22-bdd339728f14","Type":"ContainerStarted","Data":"27a274b582b2e7d7eda019a87ad3db90700fbc2f76b04024ab464d6559aa7433"}
Oct 30 21:39:46.627825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:46.627783    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" event={"ID":"02d09b60-e573-40c6-9d22-bdd339728f14","Type":"ContainerStarted","Data":"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"}
Oct 30 21:39:46.628360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:46.627982    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:46.668780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:46.668732    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" podStartSLOduration=2.328658077 podStartE2EDuration="2.668719011s" totalImagesPullingTime="340.060934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.667286648 +0000 UTC m=+2330.078826087" watchObservedRunningTime="2026-10-30 21:39:46.668719011 +0000 UTC m=+2330.080258479"
Oct 30 21:39:51.338954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.338895    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" podUID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" containerName="main" containerID="cri-o://633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0" gracePeriod=45
Oct 30 21:39:51.623003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.622977    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:39:51.667362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.667325    7037 generic.go:356] "Generic (PLEG): container finished" podID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" containerID="633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0" exitCode=0
Oct 30 21:39:51.667571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.667375    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"
Oct 30 21:39:51.667571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.667562    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" event={"ID":"47ccefd4-6668-4fd1-b51a-db4476bd2b08","Type":"ContainerDied","Data":"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"}
Oct 30 21:39:51.667702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.667595    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2" event={"ID":"47ccefd4-6668-4fd1-b51a-db4476bd2b08","Type":"ContainerDied","Data":"3c08169cd9af8b3d120d96094efe096ecc6d9f73e69d3650b23209745ccdf503"}
Oct 30 21:39:51.667702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.667617    7037 scope.go:122] "RemoveContainer" containerID="633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"
Oct 30 21:39:51.678417 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.678391    7037 scope.go:122] "RemoveContainer" containerID="633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"
Oct 30 21:39:51.678759 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:51.678731    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0\": container with ID starting with 633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0 not found: ID does not exist" containerID="633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"
Oct 30 21:39:51.678874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.678767    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0"} err="failed to get container status \"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0\": rpc error: code = NotFound desc = could not find container \"633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0\": container with ID starting with 633aa31ccb9253846aaeaadefcbaef53f9c0031deea10503829441c68df0f7c0 not found: ID does not exist"
Oct 30 21:39:51.787090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787058    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\" (UniqueName: \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787136    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787198    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787271    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787457    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787472    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:51.787552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787530    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:51.787660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787600    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\" (UniqueName: \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\") pod \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\" (UID: \"47ccefd4-6668-4fd1-b51a-db4476bd2b08\") "
Oct 30 21:39:51.787743 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.787716    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:51.788117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.788096    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.788117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.788116    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.788288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.788124    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.789379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.789355    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:51.789600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.789565    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "kube-api-access-jvff5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:51.789712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.789660    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs" pod "47ccefd4-6668-4fd1-b51a-db4476bd2b08" (UID: "47ccefd4-6668-4fd1-b51a-db4476bd2b08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:51.889383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.889302    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ccefd4-6668-4fd1-b51a-db4476bd2b08-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.889383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.889330    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ccefd4-6668-4fd1-b51a-db4476bd2b08-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.889383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.889339    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvff5\" (UniqueName: \"kubernetes.io/projected/47ccefd4-6668-4fd1-b51a-db4476bd2b08-kube-api-access-jvff5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.925267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.925227    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:39:51.925459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:51.925433    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" podUID="389efca7-cc35-480d-8e07-d857c497b058" containerName="maas-api" containerID="cri-o://fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439" gracePeriod=30
Oct 30 21:39:52.019474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.019440    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:39:52.023102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.023078    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ad73a4/auth-test-model-ad73a4-kserve-6b448b4bb9-265w2"]
Oct 30 21:39:52.199542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.199479    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:39:52.294654 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.294614    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\") pod \"389efca7-cc35-480d-8e07-d857c497b058\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") "
Oct 30 21:39:52.294864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.294737    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") pod \"389efca7-cc35-480d-8e07-d857c497b058\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") "
Oct 30 21:39:52.294864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.294821    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\") pod \"389efca7-cc35-480d-8e07-d857c497b058\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") "
Oct 30 21:39:52.295003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.294982    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\" (UniqueName: \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\") pod \"389efca7-cc35-480d-8e07-d857c497b058\" (UID: \"389efca7-cc35-480d-8e07-d857c497b058\") "
Oct 30 21:39:52.295213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.295178    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca" pod "389efca7-cc35-480d-8e07-d857c497b058" (UID: "389efca7-cc35-480d-8e07-d857c497b058"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:52.295614 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.295589    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389efca7-cc35-480d-8e07-d857c497b058-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:52.297185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.297160    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp" pod "389efca7-cc35-480d-8e07-d857c497b058" (UID: "389efca7-cc35-480d-8e07-d857c497b058"). InnerVolumeSpecName "kube-api-access-xz2sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:52.297185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.297178    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls" pod "389efca7-cc35-480d-8e07-d857c497b058" (UID: "389efca7-cc35-480d-8e07-d857c497b058"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:52.297394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.297373    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs" pod "389efca7-cc35-480d-8e07-d857c497b058" (UID: "389efca7-cc35-480d-8e07-d857c497b058"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:52.416775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.416736    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:52.416775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.416766    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389efca7-cc35-480d-8e07-d857c497b058-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:52.416775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.416777    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2sp\" (UniqueName: \"kubernetes.io/projected/389efca7-cc35-480d-8e07-d857c497b058-kube-api-access-xz2sp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:39:52.554434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.554339    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:39:52.642105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.642076    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:39:52.678057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.678020    7037 generic.go:356] "Generic (PLEG): container finished" podID="389efca7-cc35-480d-8e07-d857c497b058" containerID="fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439" exitCode=0
Oct 30 21:39:52.678238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.678074    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"
Oct 30 21:39:52.678238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.678106    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" event={"ID":"389efca7-cc35-480d-8e07-d857c497b058","Type":"ContainerDied","Data":"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"}
Oct 30 21:39:52.678238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.678163    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg" event={"ID":"389efca7-cc35-480d-8e07-d857c497b058","Type":"ContainerDied","Data":"828a426ae628c28015cab12d5b6c4ef5eff43b5d8910169d30aac24a9cc991af"}
Oct 30 21:39:52.678238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.678184    7037 scope.go:122] "RemoveContainer" containerID="fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"
Oct 30 21:39:52.689879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.689858    7037 scope.go:122] "RemoveContainer" containerID="fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"
Oct 30 21:39:52.690134 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:39:52.690114    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439\": container with ID starting with fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439 not found: ID does not exist" containerID="fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"
Oct 30 21:39:52.690208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.690142    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439"} err="failed to get container status \"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439\": rpc error: code = NotFound desc = could not find container \"fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439\": container with ID starting with fecc9b16db12436db5bb600e96dbca63f131f381ed5ce8291439b7d911e03439 not found: ID does not exist"
Oct 30 21:39:52.728588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.728552    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" path="/var/lib/kubelet/pods/47ccefd4-6668-4fd1-b51a-db4476bd2b08/volumes"
Oct 30 21:39:52.739601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.739379    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:39:52.746431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:52.746397    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f23eae-5c985fc66f-8q8qg"]
Oct 30 21:39:54.613289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:54.613257    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:39:54.708725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:39:54.708693    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389efca7-cc35-480d-8e07-d857c497b058" path="/var/lib/kubelet/pods/389efca7-cc35-480d-8e07-d857c497b058/volumes"
Oct 30 21:40:01.198048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.198019    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:40:01.228827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.228795    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\") pod \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") "
Oct 30 21:40:01.229014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.228948    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\" (UniqueName: \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\") pod \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\" (UID: \"d8b9c731-713e-44f4-a050-b7ce234e6fa8\") "
Oct 30 21:40:01.229217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.229193    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume" pod "d8b9c731-713e-44f4-a050-b7ce234e6fa8" (UID: "d8b9c731-713e-44f4-a050-b7ce234e6fa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:01.229443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.229426    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b9c731-713e-44f4-a050-b7ce234e6fa8-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:40:01.231125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.231098    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw" pod "d8b9c731-713e-44f4-a050-b7ce234e6fa8" (UID: "d8b9c731-713e-44f4-a050-b7ce234e6fa8"). InnerVolumeSpecName "kube-api-access-km9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:01.330875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.330826    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km9lw\" (UniqueName: \"kubernetes.io/projected/d8b9c731-713e-44f4-a050-b7ce234e6fa8-kube-api-access-km9lw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:40:01.759042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.759007    7037 generic.go:356] "Generic (PLEG): container finished" podID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerID="f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc" exitCode=0
Oct 30 21:40:01.759242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.759058    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"
Oct 30 21:40:01.759242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.759084    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" event={"ID":"d8b9c731-713e-44f4-a050-b7ce234e6fa8","Type":"ContainerDied","Data":"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"}
Oct 30 21:40:01.759242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.759139    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm" event={"ID":"d8b9c731-713e-44f4-a050-b7ce234e6fa8","Type":"ContainerDied","Data":"c2110296707b5fe53f1096ef4d1bcfb5ae3c2700ef8be17e381b24480ea3f0f3"}
Oct 30 21:40:01.759242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.759161    7037 scope.go:122] "RemoveContainer" containerID="f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"
Oct 30 21:40:01.776178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.776149    7037 scope.go:122] "RemoveContainer" containerID="f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"
Oct 30 21:40:01.776459 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:40:01.776439    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc\": container with ID starting with f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc not found: ID does not exist" containerID="f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"
Oct 30 21:40:01.776570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.776465    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc"} err="failed to get container status \"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc\": rpc error: code = NotFound desc = could not find container \"f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc\": container with ID starting with f3d59a6847cbaaf3953d616412f987ec95a146404f58db54e44bc7913ba2becc not found: ID does not exist"
Oct 30 21:40:01.812115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.812079    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:40:01.824649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:01.824618    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f23eae-68dd4785d7-7rccm"]
Oct 30 21:40:02.709127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:02.709086    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" path="/var/lib/kubelet/pods/d8b9c731-713e-44f4-a050-b7ce234e6fa8/volumes"
Oct 30 21:40:03.672225 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:40:03.672186    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:40:06.270306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.270226    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:40:06.270984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.270966    7037 cpu_manager.go:414] "containerMap: removing container" podUID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" containerName="main"
Oct 30 21:40:06.271045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.270985    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" containerName="main"
Oct 30 21:40:06.271045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271010    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing"
Oct 30 21:40:06.271045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271016    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing"
Oct 30 21:40:06.271045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271042    7037 cpu_manager.go:414] "containerMap: removing container" podUID="389efca7-cc35-480d-8e07-d857c497b058" containerName="maas-api"
Oct 30 21:40:06.271172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271048    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389efca7-cc35-480d-8e07-d857c497b058" containerName="maas-api"
Oct 30 21:40:06.271172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271155    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ccefd4-6668-4fd1-b51a-db4476bd2b08" containerName="main"
Oct 30 21:40:06.271172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271163    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b9c731-713e-44f4-a050-b7ce234e6fa8" containerName="payload-pre-processing"
Oct 30 21:40:06.271266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.271174    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="389efca7-cc35-480d-8e07-d857c497b058" containerName="maas-api"
Oct 30 21:40:06.276742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.276534    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.299590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.299565    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-55c5e0\""
Oct 30 21:40:06.299742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.299661    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-55c5e0-dockercfg-n7nwg\""
Oct 30 21:40:06.299880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.299862    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:40:06.407558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.399413    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwll\" (UniqueName: \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.407558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.399550    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.501066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.501032    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.501259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.501230    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwll\" (UniqueName: \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.501781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.501758    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.521313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.521256    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwll\" (UniqueName: \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\") pod \"payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.591350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.591323    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:06.900809 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:40:06.900779    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8b7bcb_cc21_4d8f_94ad_1991eb46ab83.slice/crio-9315e738ab56e006ab2ab2e993936a2f308c289e225ee6a0bae70892ee55b46d WatchSource:0}: Error finding container 9315e738ab56e006ab2ab2e993936a2f308c289e225ee6a0bae70892ee55b46d: Status 404 returned error can't find the container with id 9315e738ab56e006ab2ab2e993936a2f308c289e225ee6a0bae70892ee55b46d
Oct 30 21:40:06.901843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:06.901818    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:40:07.812484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:07.812445    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" event={"ID":"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83","Type":"ContainerStarted","Data":"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"}
Oct 30 21:40:07.812961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:07.812527    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:07.812961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:07.812553    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" event={"ID":"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83","Type":"ContainerStarted","Data":"9315e738ab56e006ab2ab2e993936a2f308c289e225ee6a0bae70892ee55b46d"}
Oct 30 21:40:07.853146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:07.853091    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" podStartSLOduration=1.8530769230000002 podStartE2EDuration="1.853076923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:07.851435056 +0000 UTC m=+2351.262974492" watchObservedRunningTime="2026-10-30 21:40:07.853076923 +0000 UTC m=+2351.264616358"
Oct 30 21:40:08.950835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.950745    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:40:08.955342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.955310    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:08.985546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.985501    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c46397\"/\"openshift-service-ca.crt\""
Oct 30 21:40:08.985546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.985522    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c46397\"/\"kube-root-ca.crt\""
Oct 30 21:40:08.985546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.985544    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c46397\"/\"default-dockercfg-tcq5t\""
Oct 30 21:40:08.985879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.985718    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c46397\"/\"auth-test-model-c46397-kserve-self-signed-certs\""
Oct 30 21:40:08.991095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:08.991065    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:40:09.033964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.033930    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpld\" (UniqueName: \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.034136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.034097    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.034196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.034137    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.034196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.034166    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.034312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.034286    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.034373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.034337    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136112    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136155    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136189    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136213    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136383 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136320    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpld\" (UniqueName: \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136652    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136654    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.136908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136886    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.137014 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.136902    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.138563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.138541    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.138882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.138861    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.155300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.155277    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpld\" (UniqueName: \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\") pod \"auth-test-model-c46397-kserve-748fc57b9c-sft4n\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") " pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.277651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.277551    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:09.582399 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:40:09.582365    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e3219e_0333_433c_97b6_94d038ca0662.slice/crio-bf6298e458e75426857e694d78d31ee5f52fd764b9c83dd5f31917e3e3504f3d WatchSource:0}: Error finding container bf6298e458e75426857e694d78d31ee5f52fd764b9c83dd5f31917e3e3504f3d: Status 404 returned error can't find the container with id bf6298e458e75426857e694d78d31ee5f52fd764b9c83dd5f31917e3e3504f3d
Oct 30 21:40:09.582550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.582468    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:40:09.829719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.829611    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" event={"ID":"86e3219e-0333-433c-97b6-94d038ca0662","Type":"ContainerStarted","Data":"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"}
Oct 30 21:40:09.829719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.829678    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" event={"ID":"86e3219e-0333-433c-97b6-94d038ca0662","Type":"ContainerStarted","Data":"bf6298e458e75426857e694d78d31ee5f52fd764b9c83dd5f31917e3e3504f3d"}
Oct 30 21:40:09.872616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:09.872566    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" podStartSLOduration=1.872552588 podStartE2EDuration="1.872552588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.869540929 +0000 UTC m=+2353.281080364" watchObservedRunningTime="2026-10-30 21:40:09.872552588 +0000 UTC m=+2353.284092077"
Oct 30 21:40:10.192777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.192701    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:40:10.197613 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.197585    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.217006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.216978    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9cd83f-serving-cert\""
Oct 30 21:40:10.217837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.217808    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:40:10.278380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.278343    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:10.356122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.356089    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.356301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.356213    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.356301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.356243    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwwd\" (UniqueName: \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.356394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.356307    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.457972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.457867    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.457972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.457918    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwwd\" (UniqueName: \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.458359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.457992    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.458359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.458251    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.459816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.459768    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.461205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.461185    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.462015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.461993    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.479727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.479692    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwwd\" (UniqueName: \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\") pod \"maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.517374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.517337    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:10.819690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.819664    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:40:10.824267 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:40:10.824238    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55e7476_3fb6_41e5_ad6e_d7b13bc414cc.slice/crio-5c14779624195bcc471ee7b506a0c130bf55c7cbea334854f11e048886e6565a WatchSource:0}: Error finding container 5c14779624195bcc471ee7b506a0c130bf55c7cbea334854f11e048886e6565a: Status 404 returned error can't find the container with id 5c14779624195bcc471ee7b506a0c130bf55c7cbea334854f11e048886e6565a
Oct 30 21:40:10.838577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:10.838542    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" event={"ID":"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc","Type":"ContainerStarted","Data":"5c14779624195bcc471ee7b506a0c130bf55c7cbea334854f11e048886e6565a"}
Oct 30 21:40:11.847870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:11.847828    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" event={"ID":"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc","Type":"ContainerStarted","Data":"66c037b42bf728beb34fcea361787eaab7b23567735e3b24472a76b5b45529b1"}
Oct 30 21:40:11.895888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:11.895830    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" podStartSLOduration=1.5690845119999999 podStartE2EDuration="1.895816437s" totalImagesPullingTime="326.731925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.895209321 +0000 UTC m=+2355.306748756" watchObservedRunningTime="2026-10-30 21:40:11.895816437 +0000 UTC m=+2355.307355872"
Oct 30 21:40:12.854268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:12.854237    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:18.825166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:18.825138    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:40:18.866088 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:18.866059    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:40:19.278606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:19.278524    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:19.291396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:19.291374    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:19.914793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:40:19.914761    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:40:33.676178 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:40:33.676140    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:41:03.673253 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:41:03.673214    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:41:17.766804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.766766    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:41:17.772344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.772322    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:17.789843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.789818    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:41:17.798632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.798608    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-da52fd\""
Oct 30 21:41:17.798632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.798617    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-da52fd-dockercfg-c5brg\""
Oct 30 21:41:17.890176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.890091    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:41:17.896553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.896508    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:17.896731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.896628    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:41:17.932535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.932502    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf6x\" (UniqueName: \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:17.932856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:17.932815    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.036329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.036232    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.036533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.036402    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvgz\" (UniqueName: \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.036533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.036434    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.036646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.036607    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf6x\" (UniqueName: \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.037013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.036993    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.056361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.056334    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf6x\" (UniqueName: \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\") pod \"payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.087191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.087167    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:18.137511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.137458    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvgz\" (UniqueName: \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.137722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.137525    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.138626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.138603    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.177760 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.177725    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvgz\" (UniqueName: \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\") pod \"payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.229314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.229041    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:18.414448 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:41:18.414412    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cca4ca_b814_4069_baf8_0f90f7768242.slice/crio-67117eaea534e4a26b7b1534d5296fff3f05ce6073965e7cfe3c52f28fb8cfa3 WatchSource:0}: Error finding container 67117eaea534e4a26b7b1534d5296fff3f05ce6073965e7cfe3c52f28fb8cfa3: Status 404 returned error can't find the container with id 67117eaea534e4a26b7b1534d5296fff3f05ce6073965e7cfe3c52f28fb8cfa3
Oct 30 21:41:18.415103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.415081    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:41:18.555292 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:41:18.555262    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebb53be_383f_4ae5_ba21_322a2c35667f.slice/crio-909386e3ea026d5427c763cd564023717bf098a4c0fd943d11c4972133f617f1 WatchSource:0}: Error finding container 909386e3ea026d5427c763cd564023717bf098a4c0fd943d11c4972133f617f1: Status 404 returned error can't find the container with id 909386e3ea026d5427c763cd564023717bf098a4c0fd943d11c4972133f617f1
Oct 30 21:41:18.555750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:18.555726    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:41:19.334544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.334472    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" event={"ID":"8ebb53be-383f-4ae5-ba21-322a2c35667f","Type":"ContainerStarted","Data":"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"}
Oct 30 21:41:19.335004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.334571    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:19.335004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.334596    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" event={"ID":"8ebb53be-383f-4ae5-ba21-322a2c35667f","Type":"ContainerStarted","Data":"909386e3ea026d5427c763cd564023717bf098a4c0fd943d11c4972133f617f1"}
Oct 30 21:41:19.336281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.336256    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" event={"ID":"68cca4ca-b814-4069-baf8-0f90f7768242","Type":"ContainerStarted","Data":"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"}
Oct 30 21:41:19.336359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.336308    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:19.336359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.336327    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" event={"ID":"68cca4ca-b814-4069-baf8-0f90f7768242","Type":"ContainerStarted","Data":"67117eaea534e4a26b7b1534d5296fff3f05ce6073965e7cfe3c52f28fb8cfa3"}
Oct 30 21:41:19.378911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.378863    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" podStartSLOduration=2.378838347 podStartE2EDuration="2.378838347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:19.376324657 +0000 UTC m=+2422.787864095" watchObservedRunningTime="2026-10-30 21:41:19.378838347 +0000 UTC m=+2422.790377783"
Oct 30 21:41:19.423761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:19.423714    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" podStartSLOduration=2.42370151 podStartE2EDuration="2.42370151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:19.420544724 +0000 UTC m=+2422.832084159" watchObservedRunningTime="2026-10-30 21:41:19.42370151 +0000 UTC m=+2422.835240977"
Oct 30 21:41:26.065228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.065191    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:41:26.070194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.070170    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.104507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.104468    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-889195\"/\"kube-root-ca.crt\""
Oct 30 21:41:26.104507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.104484    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-889195\"/\"openshift-service-ca.crt\""
Oct 30 21:41:26.104747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.104585    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-889195\"/\"default-dockercfg-jqrdk\""
Oct 30 21:41:26.104809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.104762    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-889195\"/\"auth-test-model-889195-kserve-self-signed-certs\""
Oct 30 21:41:26.108409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.108387    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:41:26.193873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.193806    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.194030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.193902    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8kr\" (UniqueName: \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.194030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.193930    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.194123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.194064    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.194123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.194096    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.194258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.194243    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295441    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295484    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295634    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295776    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295847    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8kr\" (UniqueName: \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295881    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.295914 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295887    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.296017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295917    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.296017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.295936    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.298058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.298032    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.298224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.298198    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.315085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.315057    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8kr\" (UniqueName: \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\") pod \"auth-test-model-889195-kserve-5b8f6c949-xndmq\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") " pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.396696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.396661    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:26.706999 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:41:26.706967    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a771cb_ac1a_45c3_b55f_51a28e93ea50.slice/crio-3914a0325a4a22149167187b5d891eb16988940866cc5358633b22a4c62288de WatchSource:0}: Error finding container 3914a0325a4a22149167187b5d891eb16988940866cc5358633b22a4c62288de: Status 404 returned error can't find the container with id 3914a0325a4a22149167187b5d891eb16988940866cc5358633b22a4c62288de
Oct 30 21:41:26.711386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:26.711360    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:41:27.463796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:27.463760    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" event={"ID":"89a771cb-ac1a-45c3-b55f-51a28e93ea50","Type":"ContainerStarted","Data":"f8801c02f109287b806668228201d990fc36ee9ff682b20c61f103f355da476b"}
Oct 30 21:41:27.464173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:27.463807    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" event={"ID":"89a771cb-ac1a-45c3-b55f-51a28e93ea50","Type":"ContainerStarted","Data":"3914a0325a4a22149167187b5d891eb16988940866cc5358633b22a4c62288de"}
Oct 30 21:41:27.504113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:27.504061    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" podStartSLOduration=1.504048174 podStartE2EDuration="1.504048174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:27.503032951 +0000 UTC m=+2430.914572386" watchObservedRunningTime="2026-10-30 21:41:27.504048174 +0000 UTC m=+2430.915587608"
Oct 30 21:41:28.397269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:28.397238    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:30.345969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:30.345942    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:41:30.346579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:30.346022    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:41:33.672600 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:41:33.672560    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:41:36.397070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:36.396961    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:36.410436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:36.410410    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:36.539849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:36.539818    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:41:40.395678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.395644    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:41:40.400591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.400568    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.428177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.428141    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:41:40.436666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.436630    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9cd83f\"/\"openshift-service-ca.crt\""
Oct 30 21:41:40.436833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.436747    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9cd83f\"/\"kube-root-ca.crt\""
Oct 30 21:41:40.436908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.436879    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9cd83f\"/\"auth-test-model-9cd83f-kserve-self-signed-certs\""
Oct 30 21:41:40.436988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.436974    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9cd83f\"/\"default-dockercfg-qhgrb\""
Oct 30 21:41:40.481177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481146    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.481335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481202    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpnn\" (UniqueName: \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.481335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481287    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.481412 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481368    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.481447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481434    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.481482 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.481456    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.582862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.582810    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.582878    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpnn\" (UniqueName: \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.582908    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.582974    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.583055    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.583081    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583283 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.583226    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.583309    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.583401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.583386    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.585416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.585394    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.585769 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.585747    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.602799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.602775    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpnn\" (UniqueName: \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\") pod \"auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") " pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:40.717230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:40.717157    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:41.030361 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:41:41.030320    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc95d7451_042f_4d19_af24_92e9a5a9495a.slice/crio-3313ba78086fb3a55e9e97dea4182de79084e1218b82e3ab0cf00f7acf67905c WatchSource:0}: Error finding container 3313ba78086fb3a55e9e97dea4182de79084e1218b82e3ab0cf00f7acf67905c: Status 404 returned error can't find the container with id 3313ba78086fb3a55e9e97dea4182de79084e1218b82e3ab0cf00f7acf67905c
Oct 30 21:41:41.030811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:41.030787    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:41:41.569473 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:41.569432    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" event={"ID":"c95d7451-042f-4d19-af24-92e9a5a9495a","Type":"ContainerStarted","Data":"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"}
Oct 30 21:41:41.569944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:41.569481    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" event={"ID":"c95d7451-042f-4d19-af24-92e9a5a9495a","Type":"ContainerStarted","Data":"3313ba78086fb3a55e9e97dea4182de79084e1218b82e3ab0cf00f7acf67905c"}
Oct 30 21:41:41.613801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:41.613748    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" podStartSLOduration=1.613733895 podStartE2EDuration="1.613733895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:41.612560948 +0000 UTC m=+2445.024100384" watchObservedRunningTime="2026-10-30 21:41:41.613733895 +0000 UTC m=+2445.025273331"
Oct 30 21:41:41.717560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:41.717526    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:50.718161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:50.718124    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:50.731513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:50.731452    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:41:51.660638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:41:51.660609    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:42:03.674473 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:42:03.674426    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:42:13.985783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:13.985744    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:42:29.017679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.017638    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" podUID="91434dc0-216a-4170-83a0-21e86c3e963c" containerName="main" containerID="cri-o://ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956" gracePeriod=45
Oct 30 21:42:29.317665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.317637    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:42:29.403184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403153    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\" (UniqueName: \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403234    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403311    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403387    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403436    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\" (UniqueName: \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403535    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\") pod \"91434dc0-216a-4170-83a0-21e86c3e963c\" (UID: \"91434dc0-216a-4170-83a0-21e86c3e963c\") "
Oct 30 21:42:29.403604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403535    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:29.403826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403652    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:29.403826 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.403806    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:29.404154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.404137    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.404212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.404157    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.404212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.404166    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.405860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.405831    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:29.405992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.405894    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:29.405992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.405925    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b" pod "91434dc0-216a-4170-83a0-21e86c3e963c" (UID: "91434dc0-216a-4170-83a0-21e86c3e963c"). InnerVolumeSpecName "kube-api-access-cpm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:29.505591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.505557    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91434dc0-216a-4170-83a0-21e86c3e963c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.505591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.505584    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91434dc0-216a-4170-83a0-21e86c3e963c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.505591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.505594    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpm7b\" (UniqueName: \"kubernetes.io/projected/91434dc0-216a-4170-83a0-21e86c3e963c-kube-api-access-cpm7b\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.941812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.941774    7037 generic.go:356] "Generic (PLEG): container finished" podID="91434dc0-216a-4170-83a0-21e86c3e963c" containerID="ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956" exitCode=0
Oct 30 21:42:29.941995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.941819    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"
Oct 30 21:42:29.941995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.941859    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" event={"ID":"91434dc0-216a-4170-83a0-21e86c3e963c","Type":"ContainerDied","Data":"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"}
Oct 30 21:42:29.941995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.941907    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx" event={"ID":"91434dc0-216a-4170-83a0-21e86c3e963c","Type":"ContainerDied","Data":"3d9af9c5c858da5d30f7beb0fc5e8269f45612578bb1925c6e659f51106e79e3"}
Oct 30 21:42:29.941995 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.941931    7037 scope.go:122] "RemoveContainer" containerID="ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"
Oct 30 21:42:29.953201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.953184    7037 scope.go:122] "RemoveContainer" containerID="ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"
Oct 30 21:42:29.953514 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:42:29.953463    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956\": container with ID starting with ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956 not found: ID does not exist" containerID="ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"
Oct 30 21:42:29.953630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.953510    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956"} err="failed to get container status \"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956\": rpc error: code = NotFound desc = could not find container \"ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956\": container with ID starting with ca5674f3f0b20acc572c1ee26f7fd9fcf6dd6317726e888fb6c43308ce8a5956 not found: ID does not exist"
Oct 30 21:42:29.998454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:29.998426    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:42:30.010603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:30.010573    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a4782a/auth-test-model-a4782a-kserve-559f6b6f65-pn5wx"]
Oct 30 21:42:30.718963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:30.718923    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91434dc0-216a-4170-83a0-21e86c3e963c" path="/var/lib/kubelet/pods/91434dc0-216a-4170-83a0-21e86c3e963c/volumes"
Oct 30 21:42:33.673275 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:42:33.673230    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:42:38.201925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:38.201886    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:42:44.617099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.617067    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:42:44.617988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.617971    7037 cpu_manager.go:414] "containerMap: removing container" podUID="91434dc0-216a-4170-83a0-21e86c3e963c" containerName="main"
Oct 30 21:42:44.617988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.617989    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91434dc0-216a-4170-83a0-21e86c3e963c" containerName="main"
Oct 30 21:42:44.618172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.618160    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="91434dc0-216a-4170-83a0-21e86c3e963c" containerName="main"
Oct 30 21:42:44.623478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.623459    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.638874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.638845    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:42:44.647896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.647863    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cdcf9d\""
Oct 30 21:42:44.647896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.647867    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cdcf9d-dockercfg-t6qcw\""
Oct 30 21:42:44.727321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.727281    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7wg\" (UniqueName: \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.727692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.727655    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.829996 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.829945    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7wg\" (UniqueName: \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.830229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.830030    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.830862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.830831    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.850046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.850012    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7wg\" (UniqueName: \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:44.939151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:44.939070    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:45.249961 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:42:45.249923    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f096861_6a44_4774_8401_1c099b912ca1.slice/crio-9a789a186df55d7009421c2b1c1f8c8d83655c5946e7f3d4e1cc10bd37dfb1a9 WatchSource:0}: Error finding container 9a789a186df55d7009421c2b1c1f8c8d83655c5946e7f3d4e1cc10bd37dfb1a9: Status 404 returned error can't find the container with id 9a789a186df55d7009421c2b1c1f8c8d83655c5946e7f3d4e1cc10bd37dfb1a9
Oct 30 21:42:45.252401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:45.252377    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:42:46.065387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:46.065334    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" event={"ID":"0f096861-6a44-4774-8401-1c099b912ca1","Type":"ContainerStarted","Data":"ef950569c2188fea8a2862afbffb9744e34bd5dc62a7d1bbaee6de4ad7cc9191"}
Oct 30 21:42:46.066093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:46.065401    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" event={"ID":"0f096861-6a44-4774-8401-1c099b912ca1","Type":"ContainerStarted","Data":"9a789a186df55d7009421c2b1c1f8c8d83655c5946e7f3d4e1cc10bd37dfb1a9"}
Oct 30 21:42:46.066093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:46.065692    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:42:46.109785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:46.109736    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" podStartSLOduration=2.109722748 podStartE2EDuration="2.109722748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:46.106307474 +0000 UTC m=+2509.517846910" watchObservedRunningTime="2026-10-30 21:42:46.109722748 +0000 UTC m=+2509.521262184"
Oct 30 21:42:53.126185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.126153    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:42:53.130957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.130936    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.151961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.151924    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:42:53.158059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.158022    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-55c5e0\"/\"kube-root-ca.crt\""
Oct 30 21:42:53.158059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.158049    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-55c5e0\"/\"openshift-service-ca.crt\""
Oct 30 21:42:53.163653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.163623    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-55c5e0\"/\"rate-test-model-55c5e0-kserve-self-signed-certs\""
Oct 30 21:42:53.163823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.163713    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-55c5e0\"/\"default-dockercfg-hknmr\""
Oct 30 21:42:53.229182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.229145    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" podUID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" containerName="main" containerID="cri-o://d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9" gracePeriod=45
Oct 30 21:42:53.263447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263415    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.263578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263470    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.263696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263678    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.263732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263706    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.263787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263772    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.263847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.263834    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xt2\" (UniqueName: \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365233    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365388    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365563    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365594    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365654    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365694    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365748    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xt2\" (UniqueName: \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.365957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.365935    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.366056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.366003    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.367904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.367878    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.368388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.368367    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.399629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.394426    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xt2\" (UniqueName: \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\") pod \"rate-test-model-55c5e0-kserve-78787cdd45-p5znx\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") " pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.458531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.454693    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:53.536290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.536258    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:42:53.669202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669121    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669184    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669230    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669281    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669340    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\" (UniqueName: \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\") pod \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\" (UID: \"a32a347f-36a4-4905-a315-ce0d6aebf5f9\") "
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669396    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669484    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:53.669531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.669512    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:53.670034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.670015    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.670084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.670036    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.670084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.670045    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.671509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.671471    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:53.671647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.671603    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:53.671647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.671614    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h" pod "a32a347f-36a4-4905-a315-ce0d6aebf5f9" (UID: "a32a347f-36a4-4905-a315-ce0d6aebf5f9"). InnerVolumeSpecName "kube-api-access-sqh9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:53.771148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.771113    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32a347f-36a4-4905-a315-ce0d6aebf5f9-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.771148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.771147    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32a347f-36a4-4905-a315-ce0d6aebf5f9-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.771310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.771157    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqh9h\" (UniqueName: \"kubernetes.io/projected/a32a347f-36a4-4905-a315-ce0d6aebf5f9-kube-api-access-sqh9h\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:42:53.772697 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:42:53.772672    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e85e7e_2402_46f1_9895_bd88d474174f.slice/crio-7d4cad4f5beb77f5fd9259f66551bdf4839f9d3dc54c2201ee9fdc3be4571681 WatchSource:0}: Error finding container 7d4cad4f5beb77f5fd9259f66551bdf4839f9d3dc54c2201ee9fdc3be4571681: Status 404 returned error can't find the container with id 7d4cad4f5beb77f5fd9259f66551bdf4839f9d3dc54c2201ee9fdc3be4571681
Oct 30 21:42:53.772984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:53.772964    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:42:54.127202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.127169    7037 generic.go:356] "Generic (PLEG): container finished" podID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" containerID="d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9" exitCode=0
Oct 30 21:42:54.127702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.127214    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"
Oct 30 21:42:54.127702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.127253    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" event={"ID":"a32a347f-36a4-4905-a315-ce0d6aebf5f9","Type":"ContainerDied","Data":"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"}
Oct 30 21:42:54.127702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.127293    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h" event={"ID":"a32a347f-36a4-4905-a315-ce0d6aebf5f9","Type":"ContainerDied","Data":"d66cd7e57975014d71e2e633033c1a521178a9a1e0931508b59799495cc6f068"}
Oct 30 21:42:54.127702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.127316    7037 scope.go:122] "RemoveContainer" containerID="d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"
Oct 30 21:42:54.129108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.129082    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" event={"ID":"75e85e7e-2402-46f1-9895-bd88d474174f","Type":"ContainerStarted","Data":"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"}
Oct 30 21:42:54.129226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.129128    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" event={"ID":"75e85e7e-2402-46f1-9895-bd88d474174f","Type":"ContainerStarted","Data":"7d4cad4f5beb77f5fd9259f66551bdf4839f9d3dc54c2201ee9fdc3be4571681"}
Oct 30 21:42:54.138409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.138391    7037 scope.go:122] "RemoveContainer" containerID="d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"
Oct 30 21:42:54.138773 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:42:54.138754    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9\": container with ID starting with d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9 not found: ID does not exist" containerID="d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"
Oct 30 21:42:54.138820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.138782    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9"} err="failed to get container status \"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9\": rpc error: code = NotFound desc = could not find container \"d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9\": container with ID starting with d9284cfbee1aa6539ab72b2052b02cbedfc0feb162311e3c3195b2f2422fa3f9 not found: ID does not exist"
Oct 30 21:42:54.183699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.183669    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:42:54.195256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.195231    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-75c17d/auth-test-model-75c17d-kserve-66f8774769-jnh5h"]
Oct 30 21:42:54.223268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.223222    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" podStartSLOduration=1.223208322 podStartE2EDuration="1.223208322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:54.223086618 +0000 UTC m=+2517.634626079" watchObservedRunningTime="2026-10-30 21:42:54.223208322 +0000 UTC m=+2517.634747759"
Oct 30 21:42:54.454921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.454842    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:42:54.710911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:54.710832    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" path="/var/lib/kubelet/pods/a32a347f-36a4-4905-a315-ce0d6aebf5f9/volumes"
Oct 30 21:42:57.077523 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:42:57.077468    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:43:03.455184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:03.455150    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:43:03.468311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:03.468284    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:43:03.674696 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:43:03.674658    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:43:04.229451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:04.229420    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:43:15.319148 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.319114    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:43:15.321756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.319927    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" containerName="main"
Oct 30 21:43:15.321756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.319942    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" containerName="main"
Oct 30 21:43:15.321756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.320057    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32a347f-36a4-4905-a315-ce0d6aebf5f9" containerName="main"
Oct 30 21:43:15.323833 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.323815    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.355105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.355069    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cdcf9d-serving-cert\""
Oct 30 21:43:15.360787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.360752    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:43:15.469133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.469093    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.469343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.469216    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4k6v\" (UniqueName: \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.469343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.469291    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.469460 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.469363    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.571124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.571042    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4k6v\" (UniqueName: \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.571124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.571121    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.571336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.571185    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.571485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.571458    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.572158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.572129    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.573768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.573750    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.573862 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.573769    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.592659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.592626    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4k6v\" (UniqueName: \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\") pod \"maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.640219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.640183    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:15.945175 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:43:15.945144    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e957fd_91f6_42f3_b416_210fcd63325b.slice/crio-ac8fce9826286dd895f743d00260501711d2abed8a17204269ab02b31d8b2ba0 WatchSource:0}: Error finding container ac8fce9826286dd895f743d00260501711d2abed8a17204269ab02b31d8b2ba0: Status 404 returned error can't find the container with id ac8fce9826286dd895f743d00260501711d2abed8a17204269ab02b31d8b2ba0
Oct 30 21:43:15.946707 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:15.946683    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:43:16.302336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:16.302304    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" event={"ID":"e2e957fd-91f6-42f3-b416-210fcd63325b","Type":"ContainerStarted","Data":"ac8fce9826286dd895f743d00260501711d2abed8a17204269ab02b31d8b2ba0"}
Oct 30 21:43:17.311764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:17.311728    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" event={"ID":"e2e957fd-91f6-42f3-b416-210fcd63325b","Type":"ContainerStarted","Data":"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"}
Oct 30 21:43:17.312204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:17.311791    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:17.356411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:17.356353    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" podStartSLOduration=2.007090663 podStartE2EDuration="2.356340897s" totalImagesPullingTime="349.250234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:17.353894855 +0000 UTC m=+2540.765434291" watchObservedRunningTime="2026-10-30 21:43:17.356340897 +0000 UTC m=+2540.767880333"
Oct 30 21:43:23.325079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:23.325050    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:43:33.674348 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:43:33.674310    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:43:59.488789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:43:59.488744    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:44:03.679193 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:44:03.679151    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:44:04.489224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.489176    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" podUID="0f096861-6a44-4774-8401-1c099b912ca1" containerName="payload-pre-processing" containerID="cri-o://ef950569c2188fea8a2862afbffb9744e34bd5dc62a7d1bbaee6de4ad7cc9191" gracePeriod=25
Oct 30 21:44:04.695890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.695849    7037 generic.go:356] "Generic (PLEG): container finished" podID="0f096861-6a44-4774-8401-1c099b912ca1" containerID="ef950569c2188fea8a2862afbffb9744e34bd5dc62a7d1bbaee6de4ad7cc9191" exitCode=0
Oct 30 21:44:04.696520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.696084    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" event={"ID":"0f096861-6a44-4774-8401-1c099b912ca1","Type":"ContainerDied","Data":"ef950569c2188fea8a2862afbffb9744e34bd5dc62a7d1bbaee6de4ad7cc9191"}
Oct 30 21:44:04.776461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.776434    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:44:04.903670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.903634    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\" (UniqueName: \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\") pod \"0f096861-6a44-4774-8401-1c099b912ca1\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") "
Oct 30 21:44:04.903931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.903905    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\") pod \"0f096861-6a44-4774-8401-1c099b912ca1\" (UID: \"0f096861-6a44-4774-8401-1c099b912ca1\") "
Oct 30 21:44:04.904293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.904257    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume" pod "0f096861-6a44-4774-8401-1c099b912ca1" (UID: "0f096861-6a44-4774-8401-1c099b912ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:04.904463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.904444    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f096861-6a44-4774-8401-1c099b912ca1-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:44:04.905974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:04.905945    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg" pod "0f096861-6a44-4774-8401-1c099b912ca1" (UID: "0f096861-6a44-4774-8401-1c099b912ca1"). InnerVolumeSpecName "kube-api-access-ml7wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:05.005940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.005873    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7wg\" (UniqueName: \"kubernetes.io/projected/0f096861-6a44-4774-8401-1c099b912ca1-kube-api-access-ml7wg\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:44:05.704055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.704028    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"
Oct 30 21:44:05.704511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.704052    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx" event={"ID":"0f096861-6a44-4774-8401-1c099b912ca1","Type":"ContainerDied","Data":"9a789a186df55d7009421c2b1c1f8c8d83655c5946e7f3d4e1cc10bd37dfb1a9"}
Oct 30 21:44:05.704511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.704103    7037 scope.go:122] "RemoveContainer" containerID="ef950569c2188fea8a2862afbffb9744e34bd5dc62a7d1bbaee6de4ad7cc9191"
Oct 30 21:44:05.761744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.761716    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:44:05.773184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:05.773158    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-5cb9bc454c-p2pvx"]
Oct 30 21:44:06.709989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:06.709955    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f096861-6a44-4774-8401-1c099b912ca1" path="/var/lib/kubelet/pods/0f096861-6a44-4774-8401-1c099b912ca1/volumes"
Oct 30 21:44:33.572288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.572255    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:44:33.573070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.573054    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0f096861-6a44-4774-8401-1c099b912ca1" containerName="payload-pre-processing"
Oct 30 21:44:33.573150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.573071    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f096861-6a44-4774-8401-1c099b912ca1" containerName="payload-pre-processing"
Oct 30 21:44:33.573220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.573211    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f096861-6a44-4774-8401-1c099b912ca1" containerName="payload-pre-processing"
Oct 30 21:44:33.577013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.576996    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.598099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.598067    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:44:33.607111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.607083    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cdcf9d\""
Oct 30 21:44:33.607111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.607083    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cdcf9d-dockercfg-7wqtz\""
Oct 30 21:44:33.674112 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:44:33.674075    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:44:33.719273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.719241    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.719471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.719282    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsjt\" (UniqueName: \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.733732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.733707    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:44:33.738312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.738294    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.780214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.780175    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:44:33.821370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.821318    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.821603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.821398    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsjt\" (UniqueName: \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.821603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.821516    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndk2\" (UniqueName: \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.821739 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.821660    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.822653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.822596    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.845674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.845645    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsjt\" (UniqueName: \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\") pod \"payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.890801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.890767    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:33.923974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.923939    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndk2\" (UniqueName: \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.924117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.924033    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.924776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.924751    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:33.952439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:33.949562    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndk2\" (UniqueName: \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:34.053516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.053456    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:34.206614 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:44:34.206583    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc813cc0_12fb_4c0c_bcbd_2f81dc186bf9.slice/crio-c6fdbccb341864e6ebf301b868fd6a67f3f27857f4959fca4ec68da814712330 WatchSource:0}: Error finding container c6fdbccb341864e6ebf301b868fd6a67f3f27857f4959fca4ec68da814712330: Status 404 returned error can't find the container with id c6fdbccb341864e6ebf301b868fd6a67f3f27857f4959fca4ec68da814712330
Oct 30 21:44:34.207591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.207559    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:44:34.208635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.208617    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:44:34.366300 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:44:34.366263    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be6f2aa_e135_4083_8958_122850611c7c.slice/crio-91b306227d46545df3c07273d59cdfcb56ff584c214d9db419bcd71934093638 WatchSource:0}: Error finding container 91b306227d46545df3c07273d59cdfcb56ff584c214d9db419bcd71934093638: Status 404 returned error can't find the container with id 91b306227d46545df3c07273d59cdfcb56ff584c214d9db419bcd71934093638
Oct 30 21:44:34.368861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.368835    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:44:34.944320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.944236    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" event={"ID":"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9","Type":"ContainerStarted","Data":"c6fdbccb341864e6ebf301b868fd6a67f3f27857f4959fca4ec68da814712330"}
Oct 30 21:44:34.946856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:34.946811    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" event={"ID":"7be6f2aa-e135-4083-8958-122850611c7c","Type":"ContainerStarted","Data":"91b306227d46545df3c07273d59cdfcb56ff584c214d9db419bcd71934093638"}
Oct 30 21:44:36.966650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:36.966606    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" event={"ID":"7be6f2aa-e135-4083-8958-122850611c7c","Type":"ContainerStarted","Data":"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"}
Oct 30 21:44:36.967617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:36.967590    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:44:36.969390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:36.969362    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" event={"ID":"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9","Type":"ContainerStarted","Data":"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"}
Oct 30 21:44:36.969548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:36.969418    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:37.037619 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:37.037552    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" podStartSLOduration=2.14418482 podStartE2EDuration="4.03753205s" totalImagesPullingTime="1.89334723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:37.036760271 +0000 UTC m=+2620.448299710" watchObservedRunningTime="2026-10-30 21:44:37.03753205 +0000 UTC m=+2620.449071490"
Oct 30 21:44:47.980620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:47.980585    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:44:48.040277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:48.040207    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" podStartSLOduration=12.949348237 podStartE2EDuration="15.04018872s" totalImagesPullingTime="2.090840483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:37.117889503 +0000 UTC m=+2620.529428939" watchObservedRunningTime="2026-10-30 21:44:48.04018872 +0000 UTC m=+2631.451728160"
Oct 30 21:44:48.987208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:44:48.987179    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:45:00.327927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.327891    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:00.333896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.333867    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.355958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.355859    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:45:00.379681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.379641    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:00.385309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.385171    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:00.385606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.385283    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.391331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.391305    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:00.401160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.401125    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.425030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.425001    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:00.431397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.431370    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.441542 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441513    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkrw\" (UniqueName: \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.441671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441570    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.441671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441637    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.441795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441725    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99gk\" (UniqueName: \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.441838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441805    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbt74\" (UniqueName: \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.441874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.441862    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.449855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.448934    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:00.464748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.464704    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:00.503975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.503945    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:00.509182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.509155    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.522603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.522572    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:00.539510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.538612    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:00.543124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543095    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbt74\" (UniqueName: \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.543407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543140    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.543407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543242    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543446    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjnc\" (UniqueName: \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543516    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkrw\" (UniqueName: \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543549    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543605    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543635    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.543688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543684    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnqq\" (UniqueName: \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.544143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543717    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99gk\" (UniqueName: \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.544143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543847    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:00.544143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.543953    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.544946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.544918    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.545632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.545574    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.546203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.546178    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.574087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.574059    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkrw\" (UniqueName: \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.580066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.579992    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:00.581840 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.581782    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99gk\" (UniqueName: \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\") pod \"maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.584633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.584610    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbt74\" (UniqueName: \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.644641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644611    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.644786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644724    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjnc\" (UniqueName: \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.644786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644768    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.644913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644793    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b9h\" (UniqueName: \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.644913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644820    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnqq\" (UniqueName: \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.644913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.644901    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.645345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.645324    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.645420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.645401    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.651529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.651509    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:00.685842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.685815    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnqq\" (UniqueName: \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.686102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.686077    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjnc\" (UniqueName: \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.719267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.719241    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:00.735420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.734941    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:00.748846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.748683    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:00.750117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.750088    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b9h\" (UniqueName: \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.750210 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.750195    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.753521 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.751571    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.796556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.795794    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b9h\" (UniqueName: \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.834065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.833784    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:00.862755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.862703    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:00.978284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:00.976335    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:00.992696 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:00.992654    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99cfe8d_f88d_42b1_8765_f8e15904c47c.slice/crio-e39b98d992fd16de508f266ad9666b21749d4e0d1f87b14aaa12743b2ecdc315 WatchSource:0}: Error finding container e39b98d992fd16de508f266ad9666b21749d4e0d1f87b14aaa12743b2ecdc315: Status 404 returned error can't find the container with id e39b98d992fd16de508f266ad9666b21749d4e0d1f87b14aaa12743b2ecdc315
Oct 30 21:45:01.122383 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:01.122345    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31177269_9e41_436a_8c69_59a054e9b9aa.slice/crio-407f65e43329258625f93c8a0c28ca31166c815853d92458800c0f3dd62a0b97 WatchSource:0}: Error finding container 407f65e43329258625f93c8a0c28ca31166c815853d92458800c0f3dd62a0b97: Status 404 returned error can't find the container with id 407f65e43329258625f93c8a0c28ca31166c815853d92458800c0f3dd62a0b97
Oct 30 21:45:01.124092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.124007    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:01.157327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.157301    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerStarted","Data":"e39b98d992fd16de508f266ad9666b21749d4e0d1f87b14aaa12743b2ecdc315"}
Oct 30 21:45:01.158773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.158749    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerStarted","Data":"407f65e43329258625f93c8a0c28ca31166c815853d92458800c0f3dd62a0b97"}
Oct 30 21:45:01.203888 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:01.203857    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a84d7_ceae_4cc0_80ad_00b1ca183289.slice/crio-011daf21d6a13c55b37ee1eb2655d89ab4543822f5183ecad794df25d7ba2e5d WatchSource:0}: Error finding container 011daf21d6a13c55b37ee1eb2655d89ab4543822f5183ecad794df25d7ba2e5d: Status 404 returned error can't find the container with id 011daf21d6a13c55b37ee1eb2655d89ab4543822f5183ecad794df25d7ba2e5d
Oct 30 21:45:01.205660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.205637    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:01.234211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.234187    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:01.255091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.255068    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:01.259451 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:01.259432    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:01.326775 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:01.326737    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6b4fbf_3552_4dcc_962e_5a4dda579ac4.slice/crio-7274178ac06cb241197d92d0f0bcd45f0e62f761ea6215a0b4f0e301d266b935 WatchSource:0}: Error finding container 7274178ac06cb241197d92d0f0bcd45f0e62f761ea6215a0b4f0e301d266b935: Status 404 returned error can't find the container with id 7274178ac06cb241197d92d0f0bcd45f0e62f761ea6215a0b4f0e301d266b935
Oct 30 21:45:01.328073 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:01.328033    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80222f4_e5ee_4892_93ac_c5339bff662d.slice/crio-275d7c71fafa69121a4d62a4411a2efb2c374c08d9d82a08d8a9ae0ea1783dd4 WatchSource:0}: Error finding container 275d7c71fafa69121a4d62a4411a2efb2c374c08d9d82a08d8a9ae0ea1783dd4: Status 404 returned error can't find the container with id 275d7c71fafa69121a4d62a4411a2efb2c374c08d9d82a08d8a9ae0ea1783dd4
Oct 30 21:45:01.428535 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:45:01.428478    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b92b8_bb1e_464f_821f_b23bae3e268f.slice/crio-d1429c7b61dc7463e42f8308e28f110206773cf1da087b1a3ddb6edd418bbeb7 WatchSource:0}: Error finding container d1429c7b61dc7463e42f8308e28f110206773cf1da087b1a3ddb6edd418bbeb7: Status 404 returned error can't find the container with id d1429c7b61dc7463e42f8308e28f110206773cf1da087b1a3ddb6edd418bbeb7
Oct 30 21:45:02.168206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.168168    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerStarted","Data":"8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb"}
Oct 30 21:45:02.168364 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.168233    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerStarted","Data":"011daf21d6a13c55b37ee1eb2655d89ab4543822f5183ecad794df25d7ba2e5d"}
Oct 30 21:45:02.170024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.169990    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerStarted","Data":"bf5f533db23923a274f2cfed50a61b2a65f667ef9ac98c65bfa85af2c412c13c"}
Oct 30 21:45:02.170162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.170042    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerStarted","Data":"7274178ac06cb241197d92d0f0bcd45f0e62f761ea6215a0b4f0e301d266b935"}
Oct 30 21:45:02.171683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.171660    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerStarted","Data":"0459e0f9275ef990d90ebd97cee7a9b096be18ac04855aa91fbcc9bfc6ded301"}
Oct 30 21:45:02.171775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.171706    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerStarted","Data":"275d7c71fafa69121a4d62a4411a2efb2c374c08d9d82a08d8a9ae0ea1783dd4"}
Oct 30 21:45:02.173253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.173229    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerStarted","Data":"b8d0a7cb8c597d2f5dd7c6c0043f0dfd644dd2f5c8599d3289edc1fc548acd95"}
Oct 30 21:45:02.174756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.174732    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerStarted","Data":"1fb7d277e7f7b73e5f2b270511b45ccd50dbe936b1b1547a1b21442d5b0a06f9"}
Oct 30 21:45:02.174853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.174776    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerStarted","Data":"d1429c7b61dc7463e42f8308e28f110206773cf1da087b1a3ddb6edd418bbeb7"}
Oct 30 21:45:02.176227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.176206    7037 generic.go:356] "Generic (PLEG): container finished" podID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerID="b8c11c33d83027164441ae0eb600e84437b88092c222e190304f87b32c8b501b" exitCode=60
Oct 30 21:45:02.176310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.176257    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerDied","Data":"b8c11c33d83027164441ae0eb600e84437b88092c222e190304f87b32c8b501b"}
Oct 30 21:45:02.176432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.176417    7037 scope.go:122] "RemoveContainer" containerID="b8c11c33d83027164441ae0eb600e84437b88092c222e190304f87b32c8b501b"
Oct 30 21:45:02.212943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.212881    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" podStartSLOduration=2.212861848 podStartE2EDuration="2.212861848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.209261326 +0000 UTC m=+2645.620800773" watchObservedRunningTime="2026-10-30 21:45:02.212861848 +0000 UTC m=+2645.624401288"
Oct 30 21:45:02.251305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.251252    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" podStartSLOduration=2.251237517 podStartE2EDuration="2.251237517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.249809148 +0000 UTC m=+2645.661348584" watchObservedRunningTime="2026-10-30 21:45:02.251237517 +0000 UTC m=+2645.662776954"
Oct 30 21:45:02.294433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.294338    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" podStartSLOduration=2.294325159 podStartE2EDuration="2.294325159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.291165102 +0000 UTC m=+2645.702704540" watchObservedRunningTime="2026-10-30 21:45:02.294325159 +0000 UTC m=+2645.705864596"
Oct 30 21:45:02.376994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.376939    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" podStartSLOduration=2.376924151 podStartE2EDuration="2.376924151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.375741085 +0000 UTC m=+2645.787280523" watchObservedRunningTime="2026-10-30 21:45:02.376924151 +0000 UTC m=+2645.788463754"
Oct 30 21:45:02.423783 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:02.423725    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" podStartSLOduration=2.423706386 podStartE2EDuration="2.423706386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.4203766 +0000 UTC m=+2645.831916036" watchObservedRunningTime="2026-10-30 21:45:02.423706386 +0000 UTC m=+2645.835245824"
Oct 30 21:45:02.517109 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:02.517067    7037 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a84d7_ceae_4cc0_80ad_00b1ca183289.slice/crio-8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a84d7_ceae_4cc0_80ad_00b1ca183289.slice/crio-conmon-8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:45:03.186733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.186641    7037 generic.go:356] "Generic (PLEG): container finished" podID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerID="bf5f533db23923a274f2cfed50a61b2a65f667ef9ac98c65bfa85af2c412c13c" exitCode=60
Oct 30 21:45:03.188662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.186738    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerDied","Data":"bf5f533db23923a274f2cfed50a61b2a65f667ef9ac98c65bfa85af2c412c13c"}
Oct 30 21:45:03.188662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.187159    7037 scope.go:122] "RemoveContainer" containerID="bf5f533db23923a274f2cfed50a61b2a65f667ef9ac98c65bfa85af2c412c13c"
Oct 30 21:45:03.189204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.189180    7037 generic.go:356] "Generic (PLEG): container finished" podID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerID="0459e0f9275ef990d90ebd97cee7a9b096be18ac04855aa91fbcc9bfc6ded301" exitCode=60
Oct 30 21:45:03.189281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.189259    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerDied","Data":"0459e0f9275ef990d90ebd97cee7a9b096be18ac04855aa91fbcc9bfc6ded301"}
Oct 30 21:45:03.189567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.189540    7037 scope.go:122] "RemoveContainer" containerID="0459e0f9275ef990d90ebd97cee7a9b096be18ac04855aa91fbcc9bfc6ded301"
Oct 30 21:45:03.191264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.191199    7037 generic.go:356] "Generic (PLEG): container finished" podID="31177269-9e41-436a-8c69-59a054e9b9aa" containerID="b8d0a7cb8c597d2f5dd7c6c0043f0dfd644dd2f5c8599d3289edc1fc548acd95" exitCode=60
Oct 30 21:45:03.191264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.191239    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerDied","Data":"b8d0a7cb8c597d2f5dd7c6c0043f0dfd644dd2f5c8599d3289edc1fc548acd95"}
Oct 30 21:45:03.191547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.191527    7037 scope.go:122] "RemoveContainer" containerID="b8d0a7cb8c597d2f5dd7c6c0043f0dfd644dd2f5c8599d3289edc1fc548acd95"
Oct 30 21:45:03.193121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.193104    7037 generic.go:356] "Generic (PLEG): container finished" podID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerID="1fb7d277e7f7b73e5f2b270511b45ccd50dbe936b1b1547a1b21442d5b0a06f9" exitCode=60
Oct 30 21:45:03.193218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.193183    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerDied","Data":"1fb7d277e7f7b73e5f2b270511b45ccd50dbe936b1b1547a1b21442d5b0a06f9"}
Oct 30 21:45:03.193392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.193376    7037 scope.go:122] "RemoveContainer" containerID="1fb7d277e7f7b73e5f2b270511b45ccd50dbe936b1b1547a1b21442d5b0a06f9"
Oct 30 21:45:03.195584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.195562    7037 generic.go:356] "Generic (PLEG): container finished" podID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerID="7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e" exitCode=60
Oct 30 21:45:03.195688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.195591    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerDied","Data":"7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e"}
Oct 30 21:45:03.195688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.195645    7037 scope.go:122] "RemoveContainer" containerID="b8c11c33d83027164441ae0eb600e84437b88092c222e190304f87b32c8b501b"
Oct 30 21:45:03.195889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.195871    7037 scope.go:122] "RemoveContainer" containerID="7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e"
Oct 30 21:45:03.196110 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:03.196084    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd_odh-ai-gateway-infra(e99cfe8d-f88d-42b1-8765-f8e15904c47c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c"
Oct 30 21:45:03.197182 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.197161    7037 generic.go:356] "Generic (PLEG): container finished" podID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerID="8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb" exitCode=60
Oct 30 21:45:03.197279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.197215    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerDied","Data":"8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb"}
Oct 30 21:45:03.197543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:03.197478    7037 scope.go:122] "RemoveContainer" containerID="8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb"
Oct 30 21:45:03.673767 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:03.673733    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:45:04.206506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.206461    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerStarted","Data":"50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"}
Oct 30 21:45:04.208302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.208275    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerStarted","Data":"544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"}
Oct 30 21:45:04.210029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.210005    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerStarted","Data":"fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"}
Oct 30 21:45:04.211951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.211927    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerStarted","Data":"d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"}
Oct 30 21:45:04.214038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.214014    7037 scope.go:122] "RemoveContainer" containerID="7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e"
Oct 30 21:45:04.214299 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:04.214274    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd_odh-ai-gateway-infra(e99cfe8d-f88d-42b1-8765-f8e15904c47c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c"
Oct 30 21:45:04.216157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:04.216136    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerStarted","Data":"588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"}
Oct 30 21:45:05.225742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.225704    7037 generic.go:356] "Generic (PLEG): container finished" podID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerID="d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162" exitCode=60
Oct 30 21:45:05.226326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.225814    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerDied","Data":"d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"}
Oct 30 21:45:05.226326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.225902    7037 scope.go:122] "RemoveContainer" containerID="1fb7d277e7f7b73e5f2b270511b45ccd50dbe936b1b1547a1b21442d5b0a06f9"
Oct 30 21:45:05.226326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.226052    7037 scope.go:122] "RemoveContainer" containerID="d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"
Oct 30 21:45:05.226515 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:05.226351    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz_odh-ai-gateway-infra(bf7b92b8-bb1e-464f-821f-b23bae3e268f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f"
Oct 30 21:45:05.228122 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.228100    7037 generic.go:356] "Generic (PLEG): container finished" podID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerID="588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad" exitCode=60
Oct 30 21:45:05.228226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.228165    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerDied","Data":"588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"}
Oct 30 21:45:05.228434 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.228416    7037 scope.go:122] "RemoveContainer" containerID="588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"
Oct 30 21:45:05.228747 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:05.228713    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h_odh-ai-gateway-infra(129a84d7-ceae-4cc0-80ad-00b1ca183289)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289"
Oct 30 21:45:05.230515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.230466    7037 generic.go:356] "Generic (PLEG): container finished" podID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerID="50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629" exitCode=60
Oct 30 21:45:05.230620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.230533    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerDied","Data":"50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"}
Oct 30 21:45:05.230775 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.230761    7037 scope.go:122] "RemoveContainer" containerID="50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"
Oct 30 21:45:05.231047 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:05.231026    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm_odh-ai-gateway-infra(ca6b4fbf-3552-4dcc-962e-5a4dda579ac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4"
Oct 30 21:45:05.232666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.232643    7037 generic.go:356] "Generic (PLEG): container finished" podID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerID="544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15" exitCode=60
Oct 30 21:45:05.232766 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.232724    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerDied","Data":"544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"}
Oct 30 21:45:05.232911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.232894    7037 scope.go:122] "RemoveContainer" containerID="544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"
Oct 30 21:45:05.233152 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:05.233128    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr_odh-ai-gateway-infra(a80222f4-e5ee-4892-93ac-c5339bff662d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d"
Oct 30 21:45:05.234672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.234649    7037 generic.go:356] "Generic (PLEG): container finished" podID="31177269-9e41-436a-8c69-59a054e9b9aa" containerID="fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b" exitCode=60
Oct 30 21:45:05.234764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.234700    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerDied","Data":"fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"}
Oct 30 21:45:05.234917 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.234904    7037 scope.go:122] "RemoveContainer" containerID="fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"
Oct 30 21:45:05.235144 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:05.235124    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6_odh-ai-gateway-infra(31177269-9e41-436a-8c69-59a054e9b9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" podUID="31177269-9e41-436a-8c69-59a054e9b9aa"
Oct 30 21:45:05.240633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.240613    7037 scope.go:122] "RemoveContainer" containerID="8e50df26c2778a5f8a6db7ace83cb63dea0a84b523c78dbb5af4e15e92977dcb"
Oct 30 21:45:05.254555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.254534    7037 scope.go:122] "RemoveContainer" containerID="bf5f533db23923a274f2cfed50a61b2a65f667ef9ac98c65bfa85af2c412c13c"
Oct 30 21:45:05.268012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.267990    7037 scope.go:122] "RemoveContainer" containerID="0459e0f9275ef990d90ebd97cee7a9b096be18ac04855aa91fbcc9bfc6ded301"
Oct 30 21:45:05.288812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:05.288788    7037 scope.go:122] "RemoveContainer" containerID="b8d0a7cb8c597d2f5dd7c6c0043f0dfd644dd2f5c8599d3289edc1fc548acd95"
Oct 30 21:45:06.245281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:06.245251    7037 scope.go:122] "RemoveContainer" containerID="d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"
Oct 30 21:45:06.245816 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:06.245652    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz_odh-ai-gateway-infra(bf7b92b8-bb1e-464f-821f-b23bae3e268f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f"
Oct 30 21:45:06.247135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:06.247112    7037 scope.go:122] "RemoveContainer" containerID="588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"
Oct 30 21:45:06.247460 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:06.247433    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h_odh-ai-gateway-infra(129a84d7-ceae-4cc0-80ad-00b1ca183289)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289"
Oct 30 21:45:06.248822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:06.248805    7037 scope.go:122] "RemoveContainer" containerID="50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"
Oct 30 21:45:06.249119 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:06.249099    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm_odh-ai-gateway-infra(ca6b4fbf-3552-4dcc-962e-5a4dda579ac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4"
Oct 30 21:45:06.250415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:06.250390    7037 scope.go:122] "RemoveContainer" containerID="544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"
Oct 30 21:45:06.250734 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:06.250704    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr_odh-ai-gateway-infra(a80222f4-e5ee-4892-93ac-c5339bff662d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d"
Oct 30 21:45:06.252033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:06.252012    7037 scope.go:122] "RemoveContainer" containerID="fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"
Oct 30 21:45:06.252262 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:06.252242    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6_odh-ai-gateway-infra(31177269-9e41-436a-8c69-59a054e9b9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" podUID="31177269-9e41-436a-8c69-59a054e9b9aa"
Oct 30 21:45:15.700265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:15.700232    7037 scope.go:122] "RemoveContainer" containerID="544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"
Oct 30 21:45:16.340644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:16.340607    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerStarted","Data":"cc54c8b954127941a22b5e89f97068c7cb1cd0fa7efdedce03fcebc0d04dde0e"}
Oct 30 21:45:16.704830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:16.704802    7037 scope.go:122] "RemoveContainer" containerID="588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"
Oct 30 21:45:16.705267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:16.704984    7037 scope.go:122] "RemoveContainer" containerID="fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"
Oct 30 21:45:17.349535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.349467    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerStarted","Data":"b422e00b16fc1ef33320672c65eecdbab52036acdc91494f5dadac01a015df4e"}
Oct 30 21:45:17.351609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.351583    7037 generic.go:356] "Generic (PLEG): container finished" podID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerID="cc54c8b954127941a22b5e89f97068c7cb1cd0fa7efdedce03fcebc0d04dde0e" exitCode=60
Oct 30 21:45:17.351763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.351655    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerDied","Data":"cc54c8b954127941a22b5e89f97068c7cb1cd0fa7efdedce03fcebc0d04dde0e"}
Oct 30 21:45:17.351763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.351711    7037 scope.go:122] "RemoveContainer" containerID="544221473fc2be9d5954b4893293a5e983d13c9232e043af4fe5b12fd0e02b15"
Oct 30 21:45:17.352078 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.352058    7037 scope.go:122] "RemoveContainer" containerID="cc54c8b954127941a22b5e89f97068c7cb1cd0fa7efdedce03fcebc0d04dde0e"
Oct 30 21:45:17.352418 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:17.352391    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr_odh-ai-gateway-infra(a80222f4-e5ee-4892-93ac-c5339bff662d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d"
Oct 30 21:45:17.353754 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.353733    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerStarted","Data":"59a5ac80d46754a660c4ff99bd9ea980f903b180ec38d143607e4324b852bdca"}
Oct 30 21:45:17.408353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.408315    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:17.700889 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.700846    7037 scope.go:122] "RemoveContainer" containerID="d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"
Oct 30 21:45:17.701077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.701053    7037 scope.go:122] "RemoveContainer" containerID="7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e"
Oct 30 21:45:17.701217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:17.701198    7037 scope.go:122] "RemoveContainer" containerID="50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"
Oct 30 21:45:18.365191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.365150    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerStarted","Data":"216426f5e5a04f09ee05b51957cee21d544573302fb9d41c6b2d2de406e7e7be"}
Oct 30 21:45:18.370830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.370796    7037 generic.go:356] "Generic (PLEG): container finished" podID="31177269-9e41-436a-8c69-59a054e9b9aa" containerID="59a5ac80d46754a660c4ff99bd9ea980f903b180ec38d143607e4324b852bdca" exitCode=60
Oct 30 21:45:18.371012 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.370987    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerDied","Data":"59a5ac80d46754a660c4ff99bd9ea980f903b180ec38d143607e4324b852bdca"}
Oct 30 21:45:18.371105 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.371047    7037 scope.go:122] "RemoveContainer" containerID="fad1661dad7c75e6bd58fc4543b0cf7e73daaf0a06ed22b2915cb2eac5f1530b"
Oct 30 21:45:18.374332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.371537    7037 scope.go:122] "RemoveContainer" containerID="59a5ac80d46754a660c4ff99bd9ea980f903b180ec38d143607e4324b852bdca"
Oct 30 21:45:18.374332 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:18.371936    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6_odh-ai-gateway-infra(31177269-9e41-436a-8c69-59a054e9b9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" podUID="31177269-9e41-436a-8c69-59a054e9b9aa"
Oct 30 21:45:18.374859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.374831    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerStarted","Data":"2772638afc7c04595236ffd0be8b10c8cf25eb41195546dabdd6501ca02cf19e"}
Oct 30 21:45:18.377351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.377326    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerStarted","Data":"b63bedcc978d1b2e424304c5ed5a1818ff335e9955e497dce42e282eeb1a1ef6"}
Oct 30 21:45:18.380934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.380875    7037 generic.go:356] "Generic (PLEG): container finished" podID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerID="b422e00b16fc1ef33320672c65eecdbab52036acdc91494f5dadac01a015df4e" exitCode=60
Oct 30 21:45:18.381119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.381011    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerDied","Data":"b422e00b16fc1ef33320672c65eecdbab52036acdc91494f5dadac01a015df4e"}
Oct 30 21:45:18.381804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.381550    7037 scope.go:122] "RemoveContainer" containerID="b422e00b16fc1ef33320672c65eecdbab52036acdc91494f5dadac01a015df4e"
Oct 30 21:45:18.381995 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:18.381969    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h_odh-ai-gateway-infra(129a84d7-ceae-4cc0-80ad-00b1ca183289)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289"
Oct 30 21:45:18.399137 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.399106    7037 scope.go:122] "RemoveContainer" containerID="588548121e0f787be8fc66cae376a5f9274b234ee6425bea65f99f1f4ad796ad"
Oct 30 21:45:18.436108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.435968    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:18.528659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.528608    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" podStartSLOduration=18.52859372 podStartE2EDuration="18.52859372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:18.526681255 +0000 UTC m=+2661.938220711" watchObservedRunningTime="2026-10-30 21:45:18.52859372 +0000 UTC m=+2661.940133196"
Oct 30 21:45:18.541519 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.541478    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:18.563706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.563680    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:18.643232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.643141    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\" (UniqueName: \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\") pod \"a80222f4-e5ee-4892-93ac-c5339bff662d\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") "
Oct 30 21:45:18.643458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.643319    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\") pod \"a80222f4-e5ee-4892-93ac-c5339bff662d\" (UID: \"a80222f4-e5ee-4892-93ac-c5339bff662d\") "
Oct 30 21:45:18.643813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.643785    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca" pod "a80222f4-e5ee-4892-93ac-c5339bff662d" (UID: "a80222f4-e5ee-4892-93ac-c5339bff662d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:18.645443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.645414    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw" pod "a80222f4-e5ee-4892-93ac-c5339bff662d" (UID: "a80222f4-e5ee-4892-93ac-c5339bff662d"). InnerVolumeSpecName "kube-api-access-wbkrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:18.745652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.745620    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkrw\" (UniqueName: \"kubernetes.io/projected/a80222f4-e5ee-4892-93ac-c5339bff662d-kube-api-access-wbkrw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:18.745652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:18.745650    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80222f4-e5ee-4892-93ac-c5339bff662d-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.393626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.393587    7037 generic.go:356] "Generic (PLEG): container finished" podID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerID="216426f5e5a04f09ee05b51957cee21d544573302fb9d41c6b2d2de406e7e7be" exitCode=60
Oct 30 21:45:19.394120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.393664    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerDied","Data":"216426f5e5a04f09ee05b51957cee21d544573302fb9d41c6b2d2de406e7e7be"}
Oct 30 21:45:19.394120 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.393724    7037 scope.go:122] "RemoveContainer" containerID="50f73c059ec3d2940592001eca3081ef5bd77a27779c4cdd228f6b9fdbd3f629"
Oct 30 21:45:19.394264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.394235    7037 scope.go:122] "RemoveContainer" containerID="216426f5e5a04f09ee05b51957cee21d544573302fb9d41c6b2d2de406e7e7be"
Oct 30 21:45:19.394692 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:19.394660    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm_odh-ai-gateway-infra(ca6b4fbf-3552-4dcc-962e-5a4dda579ac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4"
Oct 30 21:45:19.395810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.395785    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr" event={"ID":"a80222f4-e5ee-4892-93ac-c5339bff662d","Type":"ContainerDied","Data":"275d7c71fafa69121a4d62a4411a2efb2c374c08d9d82a08d8a9ae0ea1783dd4"}
Oct 30 21:45:19.395928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.395883    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"
Oct 30 21:45:19.402116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.402092    7037 generic.go:356] "Generic (PLEG): container finished" podID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerID="2772638afc7c04595236ffd0be8b10c8cf25eb41195546dabdd6501ca02cf19e" exitCode=60
Oct 30 21:45:19.402229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.402176    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerDied","Data":"2772638afc7c04595236ffd0be8b10c8cf25eb41195546dabdd6501ca02cf19e"}
Oct 30 21:45:19.402666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.402643    7037 scope.go:122] "RemoveContainer" containerID="2772638afc7c04595236ffd0be8b10c8cf25eb41195546dabdd6501ca02cf19e"
Oct 30 21:45:19.403031 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:19.403003    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz_odh-ai-gateway-infra(bf7b92b8-bb1e-464f-821f-b23bae3e268f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f"
Oct 30 21:45:19.409002 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.408976    7037 generic.go:356] "Generic (PLEG): container finished" podID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerID="b63bedcc978d1b2e424304c5ed5a1818ff335e9955e497dce42e282eeb1a1ef6" exitCode=60
Oct 30 21:45:19.409121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.409100    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerDied","Data":"b63bedcc978d1b2e424304c5ed5a1818ff335e9955e497dce42e282eeb1a1ef6"}
Oct 30 21:45:19.409531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.409475    7037 scope.go:122] "RemoveContainer" containerID="b63bedcc978d1b2e424304c5ed5a1818ff335e9955e497dce42e282eeb1a1ef6"
Oct 30 21:45:19.409871 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:19.409845    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd_odh-ai-gateway-infra(e99cfe8d-f88d-42b1-8765-f8e15904c47c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c"
Oct 30 21:45:19.410963 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.410941    7037 scope.go:122] "RemoveContainer" containerID="cc54c8b954127941a22b5e89f97068c7cb1cd0fa7efdedce03fcebc0d04dde0e"
Oct 30 21:45:19.425359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.425334    7037 scope.go:122] "RemoveContainer" containerID="d268d9b6e2005db17cb4788f4d724c342f479fb9be8a970b290345e6c2e8e162"
Oct 30 21:45:19.437181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.437150    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:19.449757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.449722    7037 scope.go:122] "RemoveContainer" containerID="7279cb8d29b5772eeb8c52e07c2c5057a92ae5afdaa6998460a4f06fe5001b5e"
Oct 30 21:45:19.593748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.593711    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:19.634007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.633966    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:19.642573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.642545    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f5bef0-29889945-hl9cr"]
Oct 30 21:45:19.652546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.652477    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:19.655255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.655231    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:19.724948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.724915    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:19.762827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.762795    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\") pod \"129a84d7-ceae-4cc0-80ad-00b1ca183289\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") "
Oct 30 21:45:19.763030 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.762915    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\") pod \"31177269-9e41-436a-8c69-59a054e9b9aa\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") "
Oct 30 21:45:19.763133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763047    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\" (UniqueName: \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\") pod \"129a84d7-ceae-4cc0-80ad-00b1ca183289\" (UID: \"129a84d7-ceae-4cc0-80ad-00b1ca183289\") "
Oct 30 21:45:19.763133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763097    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\" (UniqueName: \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\") pod \"31177269-9e41-436a-8c69-59a054e9b9aa\" (UID: \"31177269-9e41-436a-8c69-59a054e9b9aa\") "
Oct 30 21:45:19.763252 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763192    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca" pod "129a84d7-ceae-4cc0-80ad-00b1ca183289" (UID: "129a84d7-ceae-4cc0-80ad-00b1ca183289"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.763354 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763331    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca" pod "31177269-9e41-436a-8c69-59a054e9b9aa" (UID: "31177269-9e41-436a-8c69-59a054e9b9aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.763923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763902    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31177269-9e41-436a-8c69-59a054e9b9aa-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.764060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.763939    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129a84d7-ceae-4cc0-80ad-00b1ca183289-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.765644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.765621    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc" pod "129a84d7-ceae-4cc0-80ad-00b1ca183289" (UID: "129a84d7-ceae-4cc0-80ad-00b1ca183289"). InnerVolumeSpecName "kube-api-access-4vjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.765742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.765657    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq" pod "31177269-9e41-436a-8c69-59a054e9b9aa" (UID: "31177269-9e41-436a-8c69-59a054e9b9aa"). InnerVolumeSpecName "kube-api-access-chnqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.865508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.865454    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjnc\" (UniqueName: \"kubernetes.io/projected/129a84d7-ceae-4cc0-80ad-00b1ca183289-kube-api-access-4vjnc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.865696 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:19.865525    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnqq\" (UniqueName: \"kubernetes.io/projected/31177269-9e41-436a-8c69-59a054e9b9aa-kube-api-access-chnqq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.420892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.420855    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"
Oct 30 21:45:20.421388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.420896    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h" event={"ID":"129a84d7-ceae-4cc0-80ad-00b1ca183289","Type":"ContainerDied","Data":"011daf21d6a13c55b37ee1eb2655d89ab4543822f5183ecad794df25d7ba2e5d"}
Oct 30 21:45:20.421388 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.420965    7037 scope.go:122] "RemoveContainer" containerID="b422e00b16fc1ef33320672c65eecdbab52036acdc91494f5dadac01a015df4e"
Oct 30 21:45:20.425931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.425854    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"
Oct 30 21:45:20.425931 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.425900    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6" event={"ID":"31177269-9e41-436a-8c69-59a054e9b9aa","Type":"ContainerDied","Data":"407f65e43329258625f93c8a0c28ca31166c815853d92458800c0f3dd62a0b97"}
Oct 30 21:45:20.434944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.434913    7037 scope.go:122] "RemoveContainer" containerID="59a5ac80d46754a660c4ff99bd9ea980f903b180ec38d143607e4324b852bdca"
Oct 30 21:45:20.595181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.595146    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:20.600804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.600778    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-889195-29889945-z4tc6"]
Oct 30 21:45:20.629948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.629890    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:20.678250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.678175    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:20.697007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.696981    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:20.700577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.700548    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:20.714605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.714576    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" path="/var/lib/kubelet/pods/31177269-9e41-436a-8c69-59a054e9b9aa/volumes"
Oct 30 21:45:20.715072 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.715057    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" path="/var/lib/kubelet/pods/a80222f4-e5ee-4892-93ac-c5339bff662d/volumes"
Oct 30 21:45:20.720103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.720081    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-95692f-29889945-cvw2h"]
Oct 30 21:45:20.777222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777192    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\") pod \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") "
Oct 30 21:45:20.777416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777305    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\" (UniqueName: \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\") pod \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\" (UID: \"bf7b92b8-bb1e-464f-821f-b23bae3e268f\") "
Oct 30 21:45:20.777479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777458    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\") pod \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") "
Oct 30 21:45:20.777583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777514    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\" (UniqueName: \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\") pod \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\" (UID: \"e99cfe8d-f88d-42b1-8765-f8e15904c47c\") "
Oct 30 21:45:20.777653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777604    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\") pod \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") "
Oct 30 21:45:20.777653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777637    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca" pod "bf7b92b8-bb1e-464f-821f-b23bae3e268f" (UID: "bf7b92b8-bb1e-464f-821f-b23bae3e268f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.777776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777639    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\" (UniqueName: \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\") pod \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\" (UID: \"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4\") "
Oct 30 21:45:20.777968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.777939    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca" pod "e99cfe8d-f88d-42b1-8765-f8e15904c47c" (UID: "e99cfe8d-f88d-42b1-8765-f8e15904c47c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.778098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.778067    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca" pod "ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" (UID: "ca6b4fbf-3552-4dcc-962e-5a4dda579ac4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.778643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.778618    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b92b8-bb1e-464f-821f-b23bae3e268f-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.778643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.778645    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99cfe8d-f88d-42b1-8765-f8e15904c47c-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.778823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.778660    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.780023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.780001    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h" pod "ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" (UID: "ca6b4fbf-3552-4dcc-962e-5a4dda579ac4"). InnerVolumeSpecName "kube-api-access-w8b9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.781056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.780834    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74" pod "e99cfe8d-f88d-42b1-8765-f8e15904c47c" (UID: "e99cfe8d-f88d-42b1-8765-f8e15904c47c"). InnerVolumeSpecName "kube-api-access-wbt74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.781162 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.781094    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk" pod "bf7b92b8-bb1e-464f-821f-b23bae3e268f" (UID: "bf7b92b8-bb1e-464f-821f-b23bae3e268f"). InnerVolumeSpecName "kube-api-access-n99gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.880300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.880216    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n99gk\" (UniqueName: \"kubernetes.io/projected/bf7b92b8-bb1e-464f-821f-b23bae3e268f-kube-api-access-n99gk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.880300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.880260    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbt74\" (UniqueName: \"kubernetes.io/projected/e99cfe8d-f88d-42b1-8765-f8e15904c47c-kube-api-access-wbt74\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.880300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:20.880280    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8b9h\" (UniqueName: \"kubernetes.io/projected/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4-kube-api-access-w8b9h\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.439925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.439898    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"
Oct 30 21:45:21.440424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.439897    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz" event={"ID":"bf7b92b8-bb1e-464f-821f-b23bae3e268f","Type":"ContainerDied","Data":"d1429c7b61dc7463e42f8308e28f110206773cf1da087b1a3ddb6edd418bbeb7"}
Oct 30 21:45:21.440424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.440040    7037 scope.go:122] "RemoveContainer" containerID="2772638afc7c04595236ffd0be8b10c8cf25eb41195546dabdd6501ca02cf19e"
Oct 30 21:45:21.441448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.441421    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"
Oct 30 21:45:21.441554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.441440    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd" event={"ID":"e99cfe8d-f88d-42b1-8765-f8e15904c47c","Type":"ContainerDied","Data":"e39b98d992fd16de508f266ad9666b21749d4e0d1f87b14aaa12743b2ecdc315"}
Oct 30 21:45:21.444218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.444200    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"
Oct 30 21:45:21.444314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.444202    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm" event={"ID":"ca6b4fbf-3552-4dcc-962e-5a4dda579ac4","Type":"ContainerDied","Data":"7274178ac06cb241197d92d0f0bcd45f0e62f761ea6215a0b4f0e301d266b935"}
Oct 30 21:45:21.452185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.452126    7037 scope.go:122] "RemoveContainer" containerID="b63bedcc978d1b2e424304c5ed5a1818ff335e9955e497dce42e282eeb1a1ef6"
Oct 30 21:45:21.462578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.462553    7037 scope.go:122] "RemoveContainer" containerID="216426f5e5a04f09ee05b51957cee21d544573302fb9d41c6b2d2de406e7e7be"
Oct 30 21:45:21.498297 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.498268    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:21.510702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.510672    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-cdcf9d-29889945-72tkz"]
Oct 30 21:45:21.553820 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.553793    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:21.559312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.559289    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-da52fd-29889945-jccdm"]
Oct 30 21:45:21.596478 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.596452    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:21.599529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:21.599504    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-75c17d-29889945-6rjpd"]
Oct 30 21:45:22.710625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:22.710583    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" path="/var/lib/kubelet/pods/129a84d7-ceae-4cc0-80ad-00b1ca183289/volumes"
Oct 30 21:45:22.711238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:22.711217    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" path="/var/lib/kubelet/pods/bf7b92b8-bb1e-464f-821f-b23bae3e268f/volumes"
Oct 30 21:45:22.711838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:22.711819    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" path="/var/lib/kubelet/pods/ca6b4fbf-3552-4dcc-962e-5a4dda579ac4/volumes"
Oct 30 21:45:22.712841 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:22.712822    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" path="/var/lib/kubelet/pods/e99cfe8d-f88d-42b1-8765-f8e15904c47c/volumes"
Oct 30 21:45:23.260416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:23.260349    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:45:24.287396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:24.287360    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:45:24.287934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:24.287633    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy" containerID="cri-o://bd344a53c5d667c91322e8c59240ae4597a4f6970f4ff6b50c315efd81c926d4" gracePeriod=30
Oct 30 21:45:29.520346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.520291    7037 generic.go:356] "Generic (PLEG): container finished" podID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerID="bd344a53c5d667c91322e8c59240ae4597a4f6970f4ff6b50c315efd81c926d4" exitCode=0
Oct 30 21:45:29.520733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.520361    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" event={"ID":"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf","Type":"ContainerDied","Data":"bd344a53c5d667c91322e8c59240ae4597a4f6970f4ff6b50c315efd81c926d4"}
Oct 30 21:45:29.566329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.566268    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:45:29.608712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608683    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.608846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608764    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.608846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608795    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.608846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608817    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.608846 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608838    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.609081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608929    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.609081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.608979    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.609189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609060    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:29.609189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609129    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.609189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609058    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:29.609189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609142    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:29.609189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609161    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\") pod \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\" (UID: \"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf\") "
Oct 30 21:45:29.609485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609157    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:29.609485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609354    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:29.609805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609778    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.609805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609806    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.609993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609820    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.609993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609833    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.609993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.609847    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.611228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.611206    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:29.611317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.611227    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:29.611317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.611232    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:29.611396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.611364    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z" pod "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" (UID: "e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf"). InnerVolumeSpecName "kube-api-access-lb86z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:29.711514 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.711460    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.711691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.711520    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.711691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.711537    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb86z\" (UniqueName: \"kubernetes.io/projected/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-kube-api-access-lb86z\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:29.711691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:29.711551    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:30.161861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.161824    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:45:30.529832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.529756    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk" event={"ID":"e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf","Type":"ContainerDied","Data":"3bf839cc381dbd9f149e3bd7bd0e10d298fb7dfee4a466a9d0bed2003ac8f0a5"}
Oct 30 21:45:30.529832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.529814    7037 scope.go:122] "RemoveContainer" containerID="bd344a53c5d667c91322e8c59240ae4597a4f6970f4ff6b50c315efd81c926d4"
Oct 30 21:45:30.530300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.529765    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"
Oct 30 21:45:30.603217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.603173    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:45:30.609763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.609731    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-p8tnk"]
Oct 30 21:45:30.713219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:30.713173    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" path="/var/lib/kubelet/pods/e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf/volumes"
Oct 30 21:45:33.673053 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:33.673015    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:45:38.286988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.286941    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerName="main" containerID="cri-o://f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab" gracePeriod=45
Oct 30 21:45:38.455335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.455296    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.217:8000/ready\": dial tcp 10.134.0.217:8000: connect: connection refused"
Oct 30 21:45:38.597263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.597231    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:45:38.614251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.614219    7037 generic.go:356] "Generic (PLEG): container finished" podID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerID="f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab" exitCode=0
Oct 30 21:45:38.614406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.614258    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"
Oct 30 21:45:38.614406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.614300    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" event={"ID":"cb9f8833-dc89-4c8d-84ba-d613cebadc1c","Type":"ContainerDied","Data":"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"}
Oct 30 21:45:38.614406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.614344    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx" event={"ID":"cb9f8833-dc89-4c8d-84ba-d613cebadc1c","Type":"ContainerDied","Data":"8eae83d4c98bb616482df42512c326b6e85315e83bbe9c4362af1632fbad91ae"}
Oct 30 21:45:38.614406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.614368    7037 scope.go:122] "RemoveContainer" containerID="f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"
Oct 30 21:45:38.624679 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.624659    7037 scope.go:122] "RemoveContainer" containerID="f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"
Oct 30 21:45:38.624973 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:38.624952    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab\": container with ID starting with f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab not found: ID does not exist" containerID="f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"
Oct 30 21:45:38.625027 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.624987    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab"} err="failed to get container status \"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab\": rpc error: code = NotFound desc = could not find container \"f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab\": container with ID starting with f7711109795ff9da0703fa9d30cbcd16498bce39ef2e6f3fd921f608fba259ab not found: ID does not exist"
Oct 30 21:45:38.744635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744601    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.744842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744703    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.744842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744755    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\" (UniqueName: \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.744842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744816    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.745017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744879    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.745017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.744990    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\") pod \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\" (UID: \"cb9f8833-dc89-4c8d-84ba-d613cebadc1c\") "
Oct 30 21:45:38.745128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745009    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:38.745188 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745134    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:38.745306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745243    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:38.745793 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745769    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.745874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745801    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.745874 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.745817    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.747189 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.747160    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:38.747301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.747186    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:38.747301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.747190    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv" pod "cb9f8833-dc89-4c8d-84ba-d613cebadc1c" (UID: "cb9f8833-dc89-4c8d-84ba-d613cebadc1c"). InnerVolumeSpecName "kube-api-access-dwszv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:38.847285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.847251    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.847285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.847278    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.847285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.847289    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwszv\" (UniqueName: \"kubernetes.io/projected/cb9f8833-dc89-4c8d-84ba-d613cebadc1c-kube-api-access-dwszv\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:38.971346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.971312    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:45:38.983144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:38.983115    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e11bef/auth-test-model-e11bef-kserve-57b9dbff89-mp4sx"]
Oct 30 21:45:40.709800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:40.709765    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" path="/var/lib/kubelet/pods/cb9f8833-dc89-4c8d-84ba-d613cebadc1c/volumes"
Oct 30 21:45:45.183669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.183622    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" podUID="86e3219e-0333-433c-97b6-94d038ca0662" containerName="main" containerID="cri-o://1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d" gracePeriod=45
Oct 30 21:45:45.490311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.490281    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:45:45.644306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644273    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.644540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644352    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.644540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644411    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.644540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644441    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\" (UniqueName: \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.644752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644602    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.644752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644648    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:45.644752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644687    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:45.644752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644713    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:45.644752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.644735    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\" (UniqueName: \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\") pod \"86e3219e-0333-433c-97b6-94d038ca0662\" (UID: \"86e3219e-0333-433c-97b6-94d038ca0662\") "
Oct 30 21:45:45.645320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.645297    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.645320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.645321    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.645501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.645334    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.646950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.646920    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:45.647216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.647196    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "kube-api-access-2bpld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:45.647304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.647270    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs" pod "86e3219e-0333-433c-97b6-94d038ca0662" (UID: "86e3219e-0333-433c-97b6-94d038ca0662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:45.676342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.676308    7037 generic.go:356] "Generic (PLEG): container finished" podID="86e3219e-0333-433c-97b6-94d038ca0662" containerID="1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d" exitCode=0
Oct 30 21:45:45.676550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.676531    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" event={"ID":"86e3219e-0333-433c-97b6-94d038ca0662","Type":"ContainerDied","Data":"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"}
Oct 30 21:45:45.676631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.676576    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n" event={"ID":"86e3219e-0333-433c-97b6-94d038ca0662","Type":"ContainerDied","Data":"bf6298e458e75426857e694d78d31ee5f52fd764b9c83dd5f31917e3e3504f3d"}
Oct 30 21:45:45.676631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.676604    7037 scope.go:122] "RemoveContainer" containerID="1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"
Oct 30 21:45:45.676762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.676748    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"
Oct 30 21:45:45.691987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.691378    7037 scope.go:122] "RemoveContainer" containerID="1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"
Oct 30 21:45:45.691987 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:45:45.691772    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d\": container with ID starting with 1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d not found: ID does not exist" containerID="1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"
Oct 30 21:45:45.691987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.691815    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d"} err="failed to get container status \"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d\": rpc error: code = NotFound desc = could not find container \"1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d\": container with ID starting with 1bff580c25fbf8f313093109678dce5fdba4e76fafe2d795d751b298ac78e98d not found: ID does not exist"
Oct 30 21:45:45.747060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.746977    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bpld\" (UniqueName: \"kubernetes.io/projected/86e3219e-0333-433c-97b6-94d038ca0662-kube-api-access-2bpld\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.747060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.747008    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e3219e-0333-433c-97b6-94d038ca0662-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.747060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.747018    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e3219e-0333-433c-97b6-94d038ca0662-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:45:45.747844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.747817    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:45:45.760868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:45.760827    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c46397/auth-test-model-c46397-kserve-748fc57b9c-sft4n"]
Oct 30 21:45:46.710508 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:45:46.710458    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e3219e-0333-433c-97b6-94d038ca0662" path="/var/lib/kubelet/pods/86e3219e-0333-433c-97b6-94d038ca0662/volumes"
Oct 30 21:46:03.672780 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:46:03.672738    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:46:13.290776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.290742    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:46:13.291484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291469    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.291484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291486    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291519    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291526    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291538    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291542    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291552    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291557    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291565    7037 cpu_manager.go:414] "containerMap: removing container" podUID="86e3219e-0333-433c-97b6-94d038ca0662" containerName="main"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291569    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e3219e-0333-433c-97b6-94d038ca0662" containerName="main"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291579    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291583    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291590    7037 cpu_manager.go:414] "containerMap: removing container" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291594    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291601    7037 cpu_manager.go:414] "containerMap: removing container" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291605    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291623    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291627    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291633    7037 cpu_manager.go:414] "containerMap: removing container" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.291630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291639    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291649    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291653    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291662    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291666    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291674    7037 cpu_manager.go:414] "containerMap: removing container" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerName="main"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291679    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerName="main"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291688    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291692    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291706    7037 cpu_manager.go:414] "containerMap: removing container" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291711    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291716    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291720    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291728    7037 cpu_manager.go:414] "containerMap: removing container" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291732    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291740    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291744    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291861    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291871    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291878    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291885    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291890    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291897    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291903    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291908    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7b92b8-bb1e-464f-821f-b23bae3e268f" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291913    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9f8833-dc89-4c8d-84ba-d613cebadc1c" containerName="main"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291919    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fb5e30-cc99-44c6-a61f-7c0d04f72aaf" containerName="istio-proxy"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291925    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291930    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291937    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291944    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291951    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291956    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:13.292207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.291962    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e3219e-0333-433c-97b6-94d038ca0662" containerName="main"
Oct 30 21:46:13.297352 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.297334    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.320198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.320166    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:46:13.328598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.328567    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-203209-serving-cert\""
Oct 30 21:46:13.444557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.444524    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.444746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.444695    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.444828 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.444816    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.444872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.444849    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9sn\" (UniqueName: \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.546469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.546387    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.546651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.546544    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.546702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.546652    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.546751 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.546699    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9sn\" (UniqueName: \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.547232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.547208    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.549062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.549039    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.549236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.549216    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.566898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.566868    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9sn\" (UniqueName: \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\") pod \"maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.613911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.613882    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:13.918311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:13.918281    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:46:13.918429 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:46:13.918301    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c568bd_dbd0_4164_9e2c_1b36b1cd90d0.slice/crio-eab2d488c4608f5aebb3a172b6bc775e6c6cfd7e9f43b59189cae56a167100da WatchSource:0}: Error finding container eab2d488c4608f5aebb3a172b6bc775e6c6cfd7e9f43b59189cae56a167100da: Status 404 returned error can't find the container with id eab2d488c4608f5aebb3a172b6bc775e6c6cfd7e9f43b59189cae56a167100da
Oct 30 21:46:14.922563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:14.922524    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" event={"ID":"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0","Type":"ContainerStarted","Data":"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"}
Oct 30 21:46:14.922958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:14.922590    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" event={"ID":"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0","Type":"ContainerStarted","Data":"eab2d488c4608f5aebb3a172b6bc775e6c6cfd7e9f43b59189cae56a167100da"}
Oct 30 21:46:14.922958 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:14.922620    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:14.963431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:14.963378    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" podStartSLOduration=1.563866316 podStartE2EDuration="1.963363963s" totalImagesPullingTime="399.497647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:14.962284531 +0000 UTC m=+2718.373823980" watchObservedRunningTime="2026-10-30 21:46:14.963363963 +0000 UTC m=+2718.374903399"
Oct 30 21:46:20.935101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:20.935068    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:46:33.673699 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:46:33.673664    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:46:57.464312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.464280    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:46:57.465046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465031    7037 cpu_manager.go:414] "containerMap: removing container" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:57.465092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465047    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:57.465129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465093    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:57.465129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465099    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99cfe8d-f88d-42b1-8765-f8e15904c47c" containerName="cleanup"
Oct 30 21:46:57.465129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465108    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:57.465129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465112    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:57.465271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465220    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="31177269-9e41-436a-8c69-59a054e9b9aa" containerName="cleanup"
Oct 30 21:46:57.465271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465227    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80222f4-e5ee-4892-93ac-c5339bff662d" containerName="cleanup"
Oct 30 21:46:57.465271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465243    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6b4fbf-3552-4dcc-962e-5a4dda579ac4" containerName="cleanup"
Oct 30 21:46:57.465271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.465250    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="129a84d7-ceae-4cc0-80ad-00b1ca183289" containerName="cleanup"
Oct 30 21:46:57.469089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.469071    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.491090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.491062    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:46:57.498222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.498200    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c3bbba\""
Oct 30 21:46:57.498615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.498586    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c3bbba-dockercfg-mbf2m\""
Oct 30 21:46:57.545576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.545534    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.545759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.545633    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vssg\" (UniqueName: \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.647333 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.647290    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.647522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.647403    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vssg\" (UniqueName: \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.648829 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.648803    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.670373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.670336    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vssg\" (UniqueName: \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\") pod \"payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:57.784615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:57.784531    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:58.094537 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:46:58.088037    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbd438c_765d_4992_9247_044709ce41a5.slice/crio-06ab4167e3e5aa37d00315d36f1a5f94d4c3832f7538f382e4af1f01564de0c6 WatchSource:0}: Error finding container 06ab4167e3e5aa37d00315d36f1a5f94d4c3832f7538f382e4af1f01564de0c6: Status 404 returned error can't find the container with id 06ab4167e3e5aa37d00315d36f1a5f94d4c3832f7538f382e4af1f01564de0c6
Oct 30 21:46:58.094537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:58.088297    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:46:58.241583 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:58.241547    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" event={"ID":"3bbd438c-765d-4992-9247-044709ce41a5","Type":"ContainerStarted","Data":"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"}
Oct 30 21:46:58.241755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:58.241629    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" event={"ID":"3bbd438c-765d-4992-9247-044709ce41a5","Type":"ContainerStarted","Data":"06ab4167e3e5aa37d00315d36f1a5f94d4c3832f7538f382e4af1f01564de0c6"}
Oct 30 21:46:58.241801 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:58.241785    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:46:58.287930 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:46:58.287870    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" podStartSLOduration=1.287853518 podStartE2EDuration="1.287853518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:58.286886881 +0000 UTC m=+2761.698426317" watchObservedRunningTime="2026-10-30 21:46:58.287853518 +0000 UTC m=+2761.699392957"
Oct 30 21:47:03.672830 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:47:03.672789    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:47:09.268516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:09.264713    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:47:33.432292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:33.432255    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:47:33.679034 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:47:33.678997    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:47:47.867391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.867306    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:47:47.872702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.872679    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:47.894272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.894240    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:47:47.894642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.894624    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e1d376-serving-cert\""
Oct 30 21:47:47.968468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.968427    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vd4p\" (UniqueName: \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:47.968678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.968517    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:47.968780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.968755    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:47.968869 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:47.968851    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.069908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.069876    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.070094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.069984    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.070094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.070021    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.070211 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.070146    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vd4p\" (UniqueName: \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.070827 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.070791    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.072655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.072631    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.072759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.072638    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.090249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.090218    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vd4p\" (UniqueName: \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\") pod \"maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.190368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.190279    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:48.454264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.454179    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerName="main" containerID="cri-o://f8801c02f109287b806668228201d990fc36ee9ff682b20c61f103f355da476b" gracePeriod=45
Oct 30 21:47:48.510747 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:47:48.510706    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6d6e3a_0722_4b2b_8272_2d52086cd8ab.slice/crio-55d98bf96d1f03e2a7be1d7fda08db4a205d4a3cd296e0d0ebff15b918407867 WatchSource:0}: Error finding container 55d98bf96d1f03e2a7be1d7fda08db4a205d4a3cd296e0d0ebff15b918407867: Status 404 returned error can't find the container with id 55d98bf96d1f03e2a7be1d7fda08db4a205d4a3cd296e0d0ebff15b918407867
Oct 30 21:47:48.511714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.511691    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:47:48.530186 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.530147    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.225:8000/ready\": dial tcp 10.134.0.225:8000: connect: connection refused"
Oct 30 21:47:48.665777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.665734    7037 generic.go:356] "Generic (PLEG): container finished" podID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerID="f8801c02f109287b806668228201d990fc36ee9ff682b20c61f103f355da476b" exitCode=0
Oct 30 21:47:48.666042 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.666009    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" event={"ID":"89a771cb-ac1a-45c3-b55f-51a28e93ea50","Type":"ContainerDied","Data":"f8801c02f109287b806668228201d990fc36ee9ff682b20c61f103f355da476b"}
Oct 30 21:47:48.667925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.667892    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" event={"ID":"da6d6e3a-0722-4b2b-8272-2d52086cd8ab","Type":"ContainerStarted","Data":"55d98bf96d1f03e2a7be1d7fda08db4a205d4a3cd296e0d0ebff15b918407867"}
Oct 30 21:47:48.725695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.725665    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:47:48.881678 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.881652    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.881758    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.881804    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\" (UniqueName: \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.881885    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\" (UniqueName: \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.882001    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.882020    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:48.882529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.882263    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:48.882529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.882371    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\") pod \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\" (UID: \"89a771cb-ac1a-45c3-b55f-51a28e93ea50\") "
Oct 30 21:47:48.882708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.882686    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:48.883337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.883312    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:48.883463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.883353    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:48.883463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.883369    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:48.884302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.884281    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:48.884819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.884789    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "kube-api-access-ps8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:48.885142 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.885107    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs" pod "89a771cb-ac1a-45c3-b55f-51a28e93ea50" (UID: "89a771cb-ac1a-45c3-b55f-51a28e93ea50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:48.985373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.985291    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a771cb-ac1a-45c3-b55f-51a28e93ea50-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:48.985373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.985319    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a771cb-ac1a-45c3-b55f-51a28e93ea50-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:48.985373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:48.985332    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8kr\" (UniqueName: \"kubernetes.io/projected/89a771cb-ac1a-45c3-b55f-51a28e93ea50-kube-api-access-ps8kr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:49.678228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.678191    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" event={"ID":"da6d6e3a-0722-4b2b-8272-2d52086cd8ab","Type":"ContainerStarted","Data":"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"}
Oct 30 21:47:49.678437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.678258    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:49.679893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.679864    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"
Oct 30 21:47:49.680021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.679863    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq" event={"ID":"89a771cb-ac1a-45c3-b55f-51a28e93ea50","Type":"ContainerDied","Data":"3914a0325a4a22149167187b5d891eb16988940866cc5358633b22a4c62288de"}
Oct 30 21:47:49.680021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.679995    7037 scope.go:122] "RemoveContainer" containerID="f8801c02f109287b806668228201d990fc36ee9ff682b20c61f103f355da476b"
Oct 30 21:47:49.735663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.735618    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" podStartSLOduration=2.456558613 podStartE2EDuration="2.735603334s" totalImagesPullingTime="279.044721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.733285677 +0000 UTC m=+2813.144825114" watchObservedRunningTime="2026-10-30 21:47:49.735603334 +0000 UTC m=+2813.147142770"
Oct 30 21:47:49.781811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.781780    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:47:49.789664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:49.789634    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-889195/auth-test-model-889195-kserve-5b8f6c949-xndmq"]
Oct 30 21:47:50.710028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:50.709990    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" path="/var/lib/kubelet/pods/89a771cb-ac1a-45c3-b55f-51a28e93ea50/volumes"
Oct 30 21:47:55.693313 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:55.693280    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:47:57.883377 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:57.883339    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:47:57.883923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:57.883588    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" podUID="e2e957fd-91f6-42f3-b416-210fcd63325b" containerName="maas-api" containerID="cri-o://6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92" gracePeriod=30
Oct 30 21:47:58.025730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.025676    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:47:58.025943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.025897    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" podUID="7be6f2aa-e135-4083-8958-122850611c7c" containerName="payload-processing" containerID="cri-o://9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc" gracePeriod=30
Oct 30 21:47:58.087218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.086264    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:47:58.087218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.086533    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" podUID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" containerName="payload-pre-processing" containerID="cri-o://918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015" gracePeriod=30
Oct 30 21:47:58.237764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.237731    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:47:58.314011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.313963    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\") pod \"e2e957fd-91f6-42f3-b416-210fcd63325b\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") "
Oct 30 21:47:58.314220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.314201    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\") pod \"e2e957fd-91f6-42f3-b416-210fcd63325b\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") "
Oct 30 21:47:58.314295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.314280    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\" (UniqueName: \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\") pod \"e2e957fd-91f6-42f3-b416-210fcd63325b\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") "
Oct 30 21:47:58.314380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.314363    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\") pod \"e2e957fd-91f6-42f3-b416-210fcd63325b\" (UID: \"e2e957fd-91f6-42f3-b416-210fcd63325b\") "
Oct 30 21:47:58.315355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.315302    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca" pod "e2e957fd-91f6-42f3-b416-210fcd63325b" (UID: "e2e957fd-91f6-42f3-b416-210fcd63325b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:58.326562 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.323075    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls" pod "e2e957fd-91f6-42f3-b416-210fcd63325b" (UID: "e2e957fd-91f6-42f3-b416-210fcd63325b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:58.338705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.337551    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v" pod "e2e957fd-91f6-42f3-b416-210fcd63325b" (UID: "e2e957fd-91f6-42f3-b416-210fcd63325b"). InnerVolumeSpecName "kube-api-access-m4k6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:58.338705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.337770    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs" pod "e2e957fd-91f6-42f3-b416-210fcd63325b" (UID: "e2e957fd-91f6-42f3-b416-210fcd63325b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:58.379515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.377213    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:47:58.416636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.416549    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\" (UniqueName: \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\") pod \"7be6f2aa-e135-4083-8958-122850611c7c\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") "
Oct 30 21:47:58.416799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.416701    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\") pod \"7be6f2aa-e135-4083-8958-122850611c7c\" (UID: \"7be6f2aa-e135-4083-8958-122850611c7c\") "
Oct 30 21:47:58.417320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.417298    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.417320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.417322    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e957fd-91f6-42f3-b416-210fcd63325b-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.417552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.417337    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4k6v\" (UniqueName: \"kubernetes.io/projected/e2e957fd-91f6-42f3-b416-210fcd63325b-kube-api-access-m4k6v\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.417552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.417350    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e957fd-91f6-42f3-b416-210fcd63325b-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.417552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.417435    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume" pod "7be6f2aa-e135-4083-8958-122850611c7c" (UID: "7be6f2aa-e135-4083-8958-122850611c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:58.419685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.419659    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2" pod "7be6f2aa-e135-4083-8958-122850611c7c" (UID: "7be6f2aa-e135-4083-8958-122850611c7c"). InnerVolumeSpecName "kube-api-access-mndk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:58.454676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.454641    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:47:58.518294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.518259    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\" (UniqueName: \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\") pod \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") "
Oct 30 21:47:58.518476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.518337    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\") pod \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\" (UID: \"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9\") "
Oct 30 21:47:58.518728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.518707    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume" pod "bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" (UID: "bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:58.519052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.519030    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.519052 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.519050    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be6f2aa-e135-4083-8958-122850611c7c-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.519256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.519061    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndk2\" (UniqueName: \"kubernetes.io/projected/7be6f2aa-e135-4083-8958-122850611c7c-kube-api-access-mndk2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.520560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.520530    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt" pod "bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" (UID: "bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9"). InnerVolumeSpecName "kube-api-access-gwsjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:58.620852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.620811    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsjt\" (UniqueName: \"kubernetes.io/projected/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9-kube-api-access-gwsjt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:47:58.754212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.754171    7037 generic.go:356] "Generic (PLEG): container finished" podID="e2e957fd-91f6-42f3-b416-210fcd63325b" containerID="6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92" exitCode=0
Oct 30 21:47:58.754424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.754270    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"
Oct 30 21:47:58.754424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.754288    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" event={"ID":"e2e957fd-91f6-42f3-b416-210fcd63325b","Type":"ContainerDied","Data":"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"}
Oct 30 21:47:58.754424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.754337    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n" event={"ID":"e2e957fd-91f6-42f3-b416-210fcd63325b","Type":"ContainerDied","Data":"ac8fce9826286dd895f743d00260501711d2abed8a17204269ab02b31d8b2ba0"}
Oct 30 21:47:58.754424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.754366    7037 scope.go:122] "RemoveContainer" containerID="6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"
Oct 30 21:47:58.756458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.756428    7037 generic.go:356] "Generic (PLEG): container finished" podID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" containerID="918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015" exitCode=0
Oct 30 21:47:58.756606 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.756471    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"
Oct 30 21:47:58.756741 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.756711    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" event={"ID":"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9","Type":"ContainerDied","Data":"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"}
Oct 30 21:47:58.756860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.756775    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8" event={"ID":"bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9","Type":"ContainerDied","Data":"c6fdbccb341864e6ebf301b868fd6a67f3f27857f4959fca4ec68da814712330"}
Oct 30 21:47:58.758834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.758808    7037 generic.go:356] "Generic (PLEG): container finished" podID="7be6f2aa-e135-4083-8958-122850611c7c" containerID="9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc" exitCode=0
Oct 30 21:47:58.758957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.758846    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" event={"ID":"7be6f2aa-e135-4083-8958-122850611c7c","Type":"ContainerDied","Data":"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"}
Oct 30 21:47:58.758957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.758910    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9" event={"ID":"7be6f2aa-e135-4083-8958-122850611c7c","Type":"ContainerDied","Data":"91b306227d46545df3c07273d59cdfcb56ff584c214d9db419bcd71934093638"}
Oct 30 21:47:58.758957 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.758859    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"
Oct 30 21:47:58.769621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.769600    7037 scope.go:122] "RemoveContainer" containerID="6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"
Oct 30 21:47:58.769956 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:47:58.769933    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92\": container with ID starting with 6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92 not found: ID does not exist" containerID="6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"
Oct 30 21:47:58.770016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.769963    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92"} err="failed to get container status \"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92\": rpc error: code = NotFound desc = could not find container \"6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92\": container with ID starting with 6c52718808c45bf0a21e2ed831c4dc3dd551fa62c2aeadb550dda5fb70baec92 not found: ID does not exist"
Oct 30 21:47:58.770016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.769983    7037 scope.go:122] "RemoveContainer" containerID="918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"
Oct 30 21:47:58.784258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.784227    7037 scope.go:122] "RemoveContainer" containerID="918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"
Oct 30 21:47:58.784969 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:47:58.784934    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015\": container with ID starting with 918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015 not found: ID does not exist" containerID="918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"
Oct 30 21:47:58.785075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.784977    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015"} err="failed to get container status \"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015\": rpc error: code = NotFound desc = could not find container \"918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015\": container with ID starting with 918d611ec487fb5a91f6509edcfe5cf19498cfa8689addb9ca4d4fd3b97e1015 not found: ID does not exist"
Oct 30 21:47:58.785075 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.785001    7037 scope.go:122] "RemoveContainer" containerID="9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"
Oct 30 21:47:58.796320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.796301    7037 scope.go:122] "RemoveContainer" containerID="9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"
Oct 30 21:47:58.796626 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:47:58.796605    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc\": container with ID starting with 9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc not found: ID does not exist" containerID="9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"
Oct 30 21:47:58.796685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.796642    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc"} err="failed to get container status \"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc\": rpc error: code = NotFound desc = could not find container \"9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc\": container with ID starting with 9be242b733aa5236518c25fe0f3bd19bf1a93453372dac608fcbb0614ac1c1cc not found: ID does not exist"
Oct 30 21:47:58.853100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.853067    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:47:58.866003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.865974    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-6bcd7474d-2dqn9"]
Oct 30 21:47:58.900164 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.900131    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:47:58.903157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.903135    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cdcf9d-59dd69766c-dfdn8"]
Oct 30 21:47:58.936154 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.936113    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:47:58.939629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:47:58.939537    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cdcf9d-5489d8d68c-wzz4n"]
Oct 30 21:48:00.709197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:00.709163    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be6f2aa-e135-4083-8958-122850611c7c" path="/var/lib/kubelet/pods/7be6f2aa-e135-4083-8958-122850611c7c/volumes"
Oct 30 21:48:00.709668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:00.709653    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" path="/var/lib/kubelet/pods/bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9/volumes"
Oct 30 21:48:00.710077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:00.710064    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e957fd-91f6-42f3-b416-210fcd63325b" path="/var/lib/kubelet/pods/e2e957fd-91f6-42f3-b416-210fcd63325b/volumes"
Oct 30 21:48:03.674192 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:48:03.674156    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:48:06.233744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.233706    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:48:06.234887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234859    7037 cpu_manager.go:414] "containerMap: removing container" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerName="main"
Oct 30 21:48:06.234887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234887    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerName="main"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234912    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7be6f2aa-e135-4083-8958-122850611c7c" containerName="payload-processing"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234921    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be6f2aa-e135-4083-8958-122850611c7c" containerName="payload-processing"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234953    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e2e957fd-91f6-42f3-b416-210fcd63325b" containerName="maas-api"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.234961    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e957fd-91f6-42f3-b416-210fcd63325b" containerName="maas-api"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235060    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" containerName="payload-pre-processing"
Oct 30 21:48:06.235121 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235071    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" containerName="payload-pre-processing"
Oct 30 21:48:06.235449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235323    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be6f2aa-e135-4083-8958-122850611c7c" containerName="payload-processing"
Oct 30 21:48:06.235449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235348    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e957fd-91f6-42f3-b416-210fcd63325b" containerName="maas-api"
Oct 30 21:48:06.235449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235411    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc813cc0-12fb-4c0c-bcbd-2f81dc186bf9" containerName="payload-pre-processing"
Oct 30 21:48:06.235449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.235435    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a771cb-ac1a-45c3-b55f-51a28e93ea50" containerName="main"
Oct 30 21:48:06.241681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.241654    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.261974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.261943    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:48:06.264871 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.264847    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f5bef0\"/\"kube-root-ca.crt\""
Oct 30 21:48:06.267563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.267471    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f5bef0\"/\"openshift-service-ca.crt\""
Oct 30 21:48:06.270999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.270980    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f5bef0\"/\"default-dockercfg-pkkbj\""
Oct 30 21:48:06.273431 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.273405    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f5bef0\"/\"sub-test-model-f5bef0-kserve-self-signed-certs\""
Oct 30 21:48:06.325502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325452    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.325692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325609    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twbf\" (UniqueName: \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.325692 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325653    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.325780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325763    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.325834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325797    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.325932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.325892    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427116 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427078    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427160    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twbf\" (UniqueName: \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427198    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427223    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427247    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427291    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427526    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427603    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.427691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.427665    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.429727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.429701    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.430322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.430302    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.448256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.448223    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twbf\" (UniqueName: \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\") pod \"sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") " pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.559673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.559581    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:06.896982 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:48:06.896954    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2eabc6d_140c_436f_9e3b_c4bbe151a06c.slice/crio-2b5137c9e9062cea1f474434bf424b7a43e9fa4fb178d0600cfef45d4426a655 WatchSource:0}: Error finding container 2b5137c9e9062cea1f474434bf424b7a43e9fa4fb178d0600cfef45d4426a655: Status 404 returned error can't find the container with id 2b5137c9e9062cea1f474434bf424b7a43e9fa4fb178d0600cfef45d4426a655
Oct 30 21:48:06.898198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:06.898174    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:48:07.846129 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:07.846089    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" event={"ID":"e2eabc6d-140c-436f-9e3b-c4bbe151a06c","Type":"ContainerStarted","Data":"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"}
Oct 30 21:48:07.846611 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:07.846138    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" event={"ID":"e2eabc6d-140c-436f-9e3b-c4bbe151a06c","Type":"ContainerStarted","Data":"2b5137c9e9062cea1f474434bf424b7a43e9fa4fb178d0600cfef45d4426a655"}
Oct 30 21:48:07.891147 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:07.891095    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" podStartSLOduration=1.89108188 podStartE2EDuration="1.89108188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:07.889636959 +0000 UTC m=+2831.301176390" watchObservedRunningTime="2026-10-30 21:48:07.89108188 +0000 UTC m=+2831.302621317"
Oct 30 21:48:08.560329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:08.560296    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:16.560589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:16.560530    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:16.578791 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:16.578756    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:16.939112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:16.939082    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:48:20.042633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.042598    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:48:20.473276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.473234    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:48:20.473511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.473466    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" podUID="3bbd438c-765d-4992-9247-044709ce41a5" containerName="payload-processing" containerID="cri-o://5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa" gracePeriod=30
Oct 30 21:48:20.818498 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.818464    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:48:20.840076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.840043    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\" (UniqueName: \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\") pod \"3bbd438c-765d-4992-9247-044709ce41a5\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") "
Oct 30 21:48:20.840266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.840161    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\") pod \"3bbd438c-765d-4992-9247-044709ce41a5\" (UID: \"3bbd438c-765d-4992-9247-044709ce41a5\") "
Oct 30 21:48:20.840567 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.840543    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume" pod "3bbd438c-765d-4992-9247-044709ce41a5" (UID: "3bbd438c-765d-4992-9247-044709ce41a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:20.840701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.840685    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbd438c-765d-4992-9247-044709ce41a5-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:20.842197 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.842176    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg" pod "3bbd438c-765d-4992-9247-044709ce41a5" (UID: "3bbd438c-765d-4992-9247-044709ce41a5"). InnerVolumeSpecName "kube-api-access-6vssg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:20.954515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.951566    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vssg\" (UniqueName: \"kubernetes.io/projected/3bbd438c-765d-4992-9247-044709ce41a5-kube-api-access-6vssg\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:20.979513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.972194    7037 generic.go:356] "Generic (PLEG): container finished" podID="3bbd438c-765d-4992-9247-044709ce41a5" containerID="5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa" exitCode=0
Oct 30 21:48:20.979513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.972533    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" event={"ID":"3bbd438c-765d-4992-9247-044709ce41a5","Type":"ContainerDied","Data":"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"}
Oct 30 21:48:20.979513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.972591    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz" event={"ID":"3bbd438c-765d-4992-9247-044709ce41a5","Type":"ContainerDied","Data":"06ab4167e3e5aa37d00315d36f1a5f94d4c3832f7538f382e4af1f01564de0c6"}
Oct 30 21:48:20.979513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.972622    7037 scope.go:122] "RemoveContainer" containerID="5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"
Oct 30 21:48:20.979513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:20.972808    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"
Oct 30 21:48:21.015163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:21.015010    7037 scope.go:122] "RemoveContainer" containerID="5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"
Oct 30 21:48:21.016290 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:48:21.016203    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa\": container with ID starting with 5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa not found: ID does not exist" containerID="5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"
Oct 30 21:48:21.016290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:21.016247    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa"} err="failed to get container status \"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa\": rpc error: code = NotFound desc = could not find container \"5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa\": container with ID starting with 5255d45100a6903a78d1173bf47df4f36556f540db015752e2b1c54315ae3baa not found: ID does not exist"
Oct 30 21:48:21.052939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:21.052864    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:48:21.055969 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:21.055942    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-76456ccc86-d86kz"]
Oct 30 21:48:22.708686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:22.708654    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bbd438c-765d-4992-9247-044709ce41a5" path="/var/lib/kubelet/pods/3bbd438c-765d-4992-9247-044709ce41a5/volumes"
Oct 30 21:48:26.859055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.859020    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:48:26.859824 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.859809    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3bbd438c-765d-4992-9247-044709ce41a5" containerName="payload-processing"
Oct 30 21:48:26.859872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.859825    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbd438c-765d-4992-9247-044709ce41a5" containerName="payload-processing"
Oct 30 21:48:26.859981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.859971    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbd438c-765d-4992-9247-044709ce41a5" containerName="payload-processing"
Oct 30 21:48:26.863705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.863688    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:26.884532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.884486    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:48:26.893249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.893212    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8964ac-dockercfg-pjpgc\""
Oct 30 21:48:26.893249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.893213    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8964ac\""
Oct 30 21:48:26.932716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.932681    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26kc\" (UniqueName: \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:26.932907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.932757    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:26.986205 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.986139    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:48:26.993844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:26.993817    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.010666 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.010632    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:48:27.034664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.034556    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:27.034960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.034937    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26kc\" (UniqueName: \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:27.035560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.035455    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:27.069057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.069028    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26kc\" (UniqueName: \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:27.136246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.136144    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.136246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.136196    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfn5l\" (UniqueName: \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.178023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.177981    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:27.238191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.238161    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.238285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.238205    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfn5l\" (UniqueName: \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.239140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.239118    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.264144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.264055    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfn5l\" (UniqueName: \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\") pod \"payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.318533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.318280    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:27.531802 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:48:27.531766    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fda7b_6ff4_4a6e_b49e_f909edcca9c8.slice/crio-d1f877c4ebf739c00ad5ea8302c16bcdf66273409492d4c517f8fe3cdc18e92a WatchSource:0}: Error finding container d1f877c4ebf739c00ad5ea8302c16bcdf66273409492d4c517f8fe3cdc18e92a: Status 404 returned error can't find the container with id d1f877c4ebf739c00ad5ea8302c16bcdf66273409492d4c517f8fe3cdc18e92a
Oct 30 21:48:27.532553 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.532520    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:48:27.665977 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:48:27.665939    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a15728c_d24e_4884_b8a5_5be024cc0514.slice/crio-2764a1eb8e58e42a20f7f76afbc398cf70808117cd36cd73ef5b4a3a1e2e73c5 WatchSource:0}: Error finding container 2764a1eb8e58e42a20f7f76afbc398cf70808117cd36cd73ef5b4a3a1e2e73c5: Status 404 returned error can't find the container with id 2764a1eb8e58e42a20f7f76afbc398cf70808117cd36cd73ef5b4a3a1e2e73c5
Oct 30 21:48:27.667135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:27.667109    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:48:28.040548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.040437    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" event={"ID":"1a15728c-d24e-4884-b8a5-5be024cc0514","Type":"ContainerStarted","Data":"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"}
Oct 30 21:48:28.040548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.040523    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:28.040548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.040550    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" event={"ID":"1a15728c-d24e-4884-b8a5-5be024cc0514","Type":"ContainerStarted","Data":"2764a1eb8e58e42a20f7f76afbc398cf70808117cd36cd73ef5b4a3a1e2e73c5"}
Oct 30 21:48:28.042152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.042118    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" event={"ID":"087fda7b-6ff4-4a6e-b49e-f909edcca9c8","Type":"ContainerStarted","Data":"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"}
Oct 30 21:48:28.042300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.042180    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:28.042300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.042202    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" event={"ID":"087fda7b-6ff4-4a6e-b49e-f909edcca9c8","Type":"ContainerStarted","Data":"d1f877c4ebf739c00ad5ea8302c16bcdf66273409492d4c517f8fe3cdc18e92a"}
Oct 30 21:48:28.086442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.086392    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" podStartSLOduration=2.086378038 podStartE2EDuration="2.086378038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:28.083096467 +0000 UTC m=+2851.494635905" watchObservedRunningTime="2026-10-30 21:48:28.086378038 +0000 UTC m=+2851.497917475"
Oct 30 21:48:28.126241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:28.126142    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" podStartSLOduration=2.126123622 podStartE2EDuration="2.126123622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:28.122979983 +0000 UTC m=+2851.534519422" watchObservedRunningTime="2026-10-30 21:48:28.126123622 +0000 UTC m=+2851.537663091"
Oct 30 21:48:33.675991 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:48:33.675955    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:48:35.066946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.066892    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" podUID="c95d7451-042f-4d19-af24-92e9a5a9495a" containerName="main" containerID="cri-o://4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845" gracePeriod=45
Oct 30 21:48:35.351074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.351046    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:48:35.359609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359578    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.359756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359639    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\" (UniqueName: \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.359837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359819    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.359935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359846    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:35.359935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359856    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.359935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359893    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.360076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.359976    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\") pod \"c95d7451-042f-4d19-af24-92e9a5a9495a\" (UID: \"c95d7451-042f-4d19-af24-92e9a5a9495a\") "
Oct 30 21:48:35.360273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.360245    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:35.360273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.360266    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:35.360663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.360643    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:35.360750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.360667    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:35.360750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.360679    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:35.361993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.361965    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:35.362171 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.362152    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:35.362232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.362167    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn" pod "c95d7451-042f-4d19-af24-92e9a5a9495a" (UID: "c95d7451-042f-4d19-af24-92e9a5a9495a"). InnerVolumeSpecName "kube-api-access-2cpnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:35.462010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.461973    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cpnn\" (UniqueName: \"kubernetes.io/projected/c95d7451-042f-4d19-af24-92e9a5a9495a-kube-api-access-2cpnn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:35.462010 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.462011    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c95d7451-042f-4d19-af24-92e9a5a9495a-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:35.462294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:35.462027    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c95d7451-042f-4d19-af24-92e9a5a9495a-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:48:36.110094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.110053    7037 generic.go:356] "Generic (PLEG): container finished" podID="c95d7451-042f-4d19-af24-92e9a5a9495a" containerID="4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845" exitCode=0
Oct 30 21:48:36.110094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.110102    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"
Oct 30 21:48:36.110732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.110133    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" event={"ID":"c95d7451-042f-4d19-af24-92e9a5a9495a","Type":"ContainerDied","Data":"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"}
Oct 30 21:48:36.110732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.110176    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k" event={"ID":"c95d7451-042f-4d19-af24-92e9a5a9495a","Type":"ContainerDied","Data":"3313ba78086fb3a55e9e97dea4182de79084e1218b82e3ab0cf00f7acf67905c"}
Oct 30 21:48:36.110732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.110197    7037 scope.go:122] "RemoveContainer" containerID="4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"
Oct 30 21:48:36.123356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.123325    7037 scope.go:122] "RemoveContainer" containerID="4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"
Oct 30 21:48:36.123683 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:48:36.123657    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845\": container with ID starting with 4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845 not found: ID does not exist" containerID="4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"
Oct 30 21:48:36.123753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.123691    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845"} err="failed to get container status \"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845\": rpc error: code = NotFound desc = could not find container \"4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845\": container with ID starting with 4ca1285e1f6c445cdbbf2db772e2962db6642b6e4592eab8b2f76c4118012845 not found: ID does not exist"
Oct 30 21:48:36.179304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.179271    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:48:36.191355 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.191324    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9cd83f/auth-test-model-9cd83f-kserve-7ffb7d4497-wk44k"]
Oct 30 21:48:36.714660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:36.714628    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95d7451-042f-4d19-af24-92e9a5a9495a" path="/var/lib/kubelet/pods/c95d7451-042f-4d19-af24-92e9a5a9495a/volumes"
Oct 30 21:48:39.053201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:39.053154    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:48:39.053861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:39.053248    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:48:41.279676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:41.279646    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:48:46.280680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:46.280617    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing" containerID="cri-o://9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1" gracePeriod=25
Oct 30 21:48:53.226591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:48:53.226550    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:03.226582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:03.226539    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:03.672777 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:49:03.672738    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:49:06.592144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.592117    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:49:06.742835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.742805    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\") pod \"0cab665c-5805-43da-a21b-095dbad3bdf7\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") "
Oct 30 21:49:06.743039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.743017    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\" (UniqueName: \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\") pod \"0cab665c-5805-43da-a21b-095dbad3bdf7\" (UID: \"0cab665c-5805-43da-a21b-095dbad3bdf7\") "
Oct 30 21:49:06.743238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.743213    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume" pod "0cab665c-5805-43da-a21b-095dbad3bdf7" (UID: "0cab665c-5805-43da-a21b-095dbad3bdf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:06.743536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.743517    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cab665c-5805-43da-a21b-095dbad3bdf7-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:06.745167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.745146    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp" pod "0cab665c-5805-43da-a21b-095dbad3bdf7" (UID: "0cab665c-5805-43da-a21b-095dbad3bdf7"). InnerVolumeSpecName "kube-api-access-hl5vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:06.844384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:06.844349    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5vp\" (UniqueName: \"kubernetes.io/projected/0cab665c-5805-43da-a21b-095dbad3bdf7-kube-api-access-hl5vp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:07.367976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.367938    7037 generic.go:356] "Generic (PLEG): container finished" podID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerID="9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1" exitCode=0
Oct 30 21:49:07.368173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.367981    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"
Oct 30 21:49:07.368173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.368022    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" event={"ID":"0cab665c-5805-43da-a21b-095dbad3bdf7","Type":"ContainerDied","Data":"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"}
Oct 30 21:49:07.368173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.368075    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx" event={"ID":"0cab665c-5805-43da-a21b-095dbad3bdf7","Type":"ContainerDied","Data":"1a62226fbb8ca2dee450eb61cf209f0b093ccc70a64f87e20f06b9842faaf30f"}
Oct 30 21:49:07.368173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.368099    7037 scope.go:122] "RemoveContainer" containerID="9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"
Oct 30 21:49:07.379731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.379708    7037 scope.go:122] "RemoveContainer" containerID="9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"
Oct 30 21:49:07.380009 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:49:07.379991    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1\": container with ID starting with 9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1 not found: ID does not exist" containerID="9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"
Oct 30 21:49:07.380055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.380016    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1"} err="failed to get container status \"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1\": rpc error: code = NotFound desc = could not find container \"9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1\": container with ID starting with 9f5c75a966ef5a6b3c94263b7816c9a832445cf3ea15a70a7b49c0355d9cd9b1 not found: ID does not exist"
Oct 30 21:49:07.420382 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.420351    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:49:07.432569 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.432541    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a4782a-fd478b978-8rkjx"]
Oct 30 21:49:07.878800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.878764    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:49:07.879597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879579    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c95d7451-042f-4d19-af24-92e9a5a9495a" containerName="main"
Oct 30 21:49:07.879672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879600    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95d7451-042f-4d19-af24-92e9a5a9495a" containerName="main"
Oct 30 21:49:07.879672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879643    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing"
Oct 30 21:49:07.879672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879649    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing"
Oct 30 21:49:07.879802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879791    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95d7451-042f-4d19-af24-92e9a5a9495a" containerName="main"
Oct 30 21:49:07.879838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.879810    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" containerName="payload-processing"
Oct 30 21:49:07.885048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.885018    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:07.909618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.909573    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8964ac-serving-cert\""
Oct 30 21:49:07.914206 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:07.914178    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:49:08.058213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.058162    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.058394 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.058365    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.058530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.058502    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.058665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.058648    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhct\" (UniqueName: \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.159770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.159681    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.159928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.159800    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.159928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.159856    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.160006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.159939    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhct\" (UniqueName: \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.160527 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.160482    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.162317 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.162289    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.162442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.162348    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.180976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.180938    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhct\" (UniqueName: \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\") pod \"maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.202224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.202192    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:08.518637 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:49:08.518606    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1178672_53f6_4fd6_a680_de16cb780487.slice/crio-0d4b3e5e5234677d32aad5493844cd6d7f7647a18ef8c6377affb7456fde94ea WatchSource:0}: Error finding container 0d4b3e5e5234677d32aad5493844cd6d7f7647a18ef8c6377affb7456fde94ea: Status 404 returned error can't find the container with id 0d4b3e5e5234677d32aad5493844cd6d7f7647a18ef8c6377affb7456fde94ea
Oct 30 21:49:08.519169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.519073    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:49:08.710071 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:08.710027    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cab665c-5805-43da-a21b-095dbad3bdf7" path="/var/lib/kubelet/pods/0cab665c-5805-43da-a21b-095dbad3bdf7/volumes"
Oct 30 21:49:09.388329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:09.388284    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" event={"ID":"d1178672-53f6-4fd6-a680-de16cb780487","Type":"ContainerStarted","Data":"7732909fce5a79a11f72911c85befd433a74011455cc2412e7a9e9a43822fc37"}
Oct 30 21:49:09.388951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:09.388346    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" event={"ID":"d1178672-53f6-4fd6-a680-de16cb780487","Type":"ContainerStarted","Data":"0d4b3e5e5234677d32aad5493844cd6d7f7647a18ef8c6377affb7456fde94ea"}
Oct 30 21:49:09.388951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:09.388374    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:09.436094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:09.436038    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" podStartSLOduration=2.046435943 podStartE2EDuration="2.436019412s" totalImagesPullingTime="389.583469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:09.433823155 +0000 UTC m=+2892.845362593" watchObservedRunningTime="2026-10-30 21:49:09.436019412 +0000 UTC m=+2892.847558849"
Oct 30 21:49:15.404556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:15.404525    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:49:28.286087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:28.286047    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:49:33.674108 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:49:33.674065    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:49:38.626144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.626112    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:49:38.631216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.631192    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.656384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.656349    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:49:38.675214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.675179    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9a8ed\""
Oct 30 21:49:38.675981 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.675934    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9a8ed-dockercfg-5wvsd\""
Oct 30 21:49:38.693641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.693603    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.694338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.694046    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq2q\" (UniqueName: \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.795338 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.795297    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.795554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.795531    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq2q\" (UniqueName: \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.796176 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.796149    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.816041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.816012    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq2q\" (UniqueName: \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\") pod \"payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:38.947909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:38.947823    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:39.249866 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:49:39.249834    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ce0c9d_a48a_4a90_b20b_91d063c8bff7.slice/crio-aae0a0443a813e7281fe983b3803f7aa808318142af0ee1f06ee28bd1812d5d1 WatchSource:0}: Error finding container aae0a0443a813e7281fe983b3803f7aa808318142af0ee1f06ee28bd1812d5d1: Status 404 returned error can't find the container with id aae0a0443a813e7281fe983b3803f7aa808318142af0ee1f06ee28bd1812d5d1
Oct 30 21:49:39.250929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:39.250904    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:49:39.621264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:39.621219    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" event={"ID":"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7","Type":"ContainerStarted","Data":"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"}
Oct 30 21:49:39.621452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:39.621288    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" event={"ID":"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7","Type":"ContainerStarted","Data":"aae0a0443a813e7281fe983b3803f7aa808318142af0ee1f06ee28bd1812d5d1"}
Oct 30 21:49:39.621452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:39.621314    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:39.661506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:39.661446    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" podStartSLOduration=1.661431392 podStartE2EDuration="1.661431392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.660646119 +0000 UTC m=+2923.072185557" watchObservedRunningTime="2026-10-30 21:49:39.661431392 +0000 UTC m=+2923.072970828"
Oct 30 21:49:43.317706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.317658    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" podUID="75e85e7e-2402-46f1-9895-bd88d474174f" containerName="main" containerID="cri-o://e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4" gracePeriod=45
Oct 30 21:49:43.609802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.609754    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:49:43.652892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.652857    7037 generic.go:356] "Generic (PLEG): container finished" podID="75e85e7e-2402-46f1-9895-bd88d474174f" containerID="e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4" exitCode=0
Oct 30 21:49:43.653091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.652916    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"
Oct 30 21:49:43.653091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.652968    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" event={"ID":"75e85e7e-2402-46f1-9895-bd88d474174f","Type":"ContainerDied","Data":"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"}
Oct 30 21:49:43.653091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.653002    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx" event={"ID":"75e85e7e-2402-46f1-9895-bd88d474174f","Type":"ContainerDied","Data":"7d4cad4f5beb77f5fd9259f66551bdf4839f9d3dc54c2201ee9fdc3be4571681"}
Oct 30 21:49:43.653091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.653023    7037 scope.go:122] "RemoveContainer" containerID="e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"
Oct 30 21:49:43.662060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.661970    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.662232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.662153    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.662303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.662228    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.662303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.662281    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\" (UniqueName: \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.662809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.662377    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.662809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.662506    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\" (UniqueName: \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\") pod \"75e85e7e-2402-46f1-9895-bd88d474174f\" (UID: \"75e85e7e-2402-46f1-9895-bd88d474174f\") "
Oct 30 21:49:43.663230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.663208    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:43.663694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.663664    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:43.663893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.663869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:43.666712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.666690    7037 scope.go:122] "RemoveContainer" containerID="e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"
Oct 30 21:49:43.667124 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:49:43.667094    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4\": container with ID starting with e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4 not found: ID does not exist" containerID="e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"
Oct 30 21:49:43.667220 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.667134    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4"} err="failed to get container status \"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4\": rpc error: code = NotFound desc = could not find container \"e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4\": container with ID starting with e8cbb475d136146de9a7f7150e8f557e02ad6d85d29de69941c433c92655ced4 not found: ID does not exist"
Oct 30 21:49:43.667406 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.667386    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:43.667661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.667639    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "kube-api-access-b8xt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:43.673043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.673017    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs" pod "75e85e7e-2402-46f1-9895-bd88d474174f" (UID: "75e85e7e-2402-46f1-9895-bd88d474174f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:43.764771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764736    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:43.764771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764767    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:43.764771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764778    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e85e7e-2402-46f1-9895-bd88d474174f-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:43.765029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764790    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:43.765029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764801    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xt2\" (UniqueName: \"kubernetes.io/projected/75e85e7e-2402-46f1-9895-bd88d474174f-kube-api-access-b8xt2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:43.765029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:43.764815    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e85e7e-2402-46f1-9895-bd88d474174f-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:49:44.002848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:44.002808    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:49:44.014997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:44.014962    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-55c5e0/rate-test-model-55c5e0-kserve-78787cdd45-p5znx"]
Oct 30 21:49:44.709314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:44.709281    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e85e7e-2402-46f1-9895-bd88d474174f" path="/var/lib/kubelet/pods/75e85e7e-2402-46f1-9895-bd88d474174f/volumes"
Oct 30 21:49:50.634050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:50.634019    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:49:57.060443 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:57.060403    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:49:57.096024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:57.095988    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:49:57.466836 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:57.466798    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:49:57.494807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:57.494774    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:49:57.502048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:49:57.502017    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:50:02.061660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.061612    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" podUID="1a15728c-d24e-4884-b8a5-5be024cc0514" containerName="payload-processing" containerID="cri-o://f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d" gracePeriod=25
Oct 30 21:50:02.097161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.097121    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" podUID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" containerName="payload-pre-processing" containerID="cri-o://05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133" gracePeriod=25
Oct 30 21:50:02.380701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.380672    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:50:02.415936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.415909    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:50:02.424117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.424086    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\" (UniqueName: \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\") pod \"1a15728c-d24e-4884-b8a5-5be024cc0514\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") "
Oct 30 21:50:02.424293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.424273    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\") pod \"1a15728c-d24e-4884-b8a5-5be024cc0514\" (UID: \"1a15728c-d24e-4884-b8a5-5be024cc0514\") "
Oct 30 21:50:02.424632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.424611    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume" pod "1a15728c-d24e-4884-b8a5-5be024cc0514" (UID: "1a15728c-d24e-4884-b8a5-5be024cc0514"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:02.424835 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.424816    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a15728c-d24e-4884-b8a5-5be024cc0514-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:02.426247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.426223    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l" pod "1a15728c-d24e-4884-b8a5-5be024cc0514" (UID: "1a15728c-d24e-4884-b8a5-5be024cc0514"). InnerVolumeSpecName "kube-api-access-wfn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:02.525337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.525298    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\" (UniqueName: \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\") pod \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") "
Oct 30 21:50:02.525579 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.525454    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\") pod \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\" (UID: \"087fda7b-6ff4-4a6e-b49e-f909edcca9c8\") "
Oct 30 21:50:02.525910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.525884    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume" pod "087fda7b-6ff4-4a6e-b49e-f909edcca9c8" (UID: "087fda7b-6ff4-4a6e-b49e-f909edcca9c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:02.525987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.525963    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:02.525987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.525980    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfn5l\" (UniqueName: \"kubernetes.io/projected/1a15728c-d24e-4884-b8a5-5be024cc0514-kube-api-access-wfn5l\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:02.527446 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.527418    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc" pod "087fda7b-6ff4-4a6e-b49e-f909edcca9c8" (UID: "087fda7b-6ff4-4a6e-b49e-f909edcca9c8"). InnerVolumeSpecName "kube-api-access-x26kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:02.627371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.627287    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26kc\" (UniqueName: \"kubernetes.io/projected/087fda7b-6ff4-4a6e-b49e-f909edcca9c8-kube-api-access-x26kc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:02.816860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.816810    7037 generic.go:356] "Generic (PLEG): container finished" podID="1a15728c-d24e-4884-b8a5-5be024cc0514" containerID="f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d" exitCode=0
Oct 30 21:50:02.817077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.816874    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"
Oct 30 21:50:02.817077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.817002    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" event={"ID":"1a15728c-d24e-4884-b8a5-5be024cc0514","Type":"ContainerDied","Data":"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"}
Oct 30 21:50:02.817077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.817049    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt" event={"ID":"1a15728c-d24e-4884-b8a5-5be024cc0514","Type":"ContainerDied","Data":"2764a1eb8e58e42a20f7f76afbc398cf70808117cd36cd73ef5b4a3a1e2e73c5"}
Oct 30 21:50:02.817077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.817078    7037 scope.go:122] "RemoveContainer" containerID="f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"
Oct 30 21:50:02.820774 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.820736    7037 generic.go:356] "Generic (PLEG): container finished" podID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" containerID="05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133" exitCode=0
Oct 30 21:50:02.820911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.820804    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"
Oct 30 21:50:02.820911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.820884    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" event={"ID":"087fda7b-6ff4-4a6e-b49e-f909edcca9c8","Type":"ContainerDied","Data":"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"}
Oct 30 21:50:02.821043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.820933    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m" event={"ID":"087fda7b-6ff4-4a6e-b49e-f909edcca9c8","Type":"ContainerDied","Data":"d1f877c4ebf739c00ad5ea8302c16bcdf66273409492d4c517f8fe3cdc18e92a"}
Oct 30 21:50:02.835062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.835035    7037 scope.go:122] "RemoveContainer" containerID="f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"
Oct 30 21:50:02.835406 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:02.835365    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d\": container with ID starting with f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d not found: ID does not exist" containerID="f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"
Oct 30 21:50:02.835524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.835413    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d"} err="failed to get container status \"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d\": rpc error: code = NotFound desc = could not find container \"f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d\": container with ID starting with f92fe60698e8aef88f6e8f68281157bae3ee66fac87857ac3b141f86fe2e319d not found: ID does not exist"
Oct 30 21:50:02.835524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.835432    7037 scope.go:122] "RemoveContainer" containerID="05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"
Oct 30 21:50:02.848712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.848681    7037 scope.go:122] "RemoveContainer" containerID="05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"
Oct 30 21:50:02.849028 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:02.849004    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133\": container with ID starting with 05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133 not found: ID does not exist" containerID="05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"
Oct 30 21:50:02.849140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.849034    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133"} err="failed to get container status \"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133\": rpc error: code = NotFound desc = could not find container \"05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133\": container with ID starting with 05147223e76bb63bf11c94e73be05cb972ebacc9619d8e04777e2ccfc09e8133 not found: ID does not exist"
Oct 30 21:50:02.877802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.877723    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:50:02.881608 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.881576    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8964ac-976f957fd-vhvrt"]
Oct 30 21:50:02.916765 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.916727    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:50:02.919708 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:02.919687    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f55c6c75f-jx66m"]
Oct 30 21:50:03.205300 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:03.205214    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:50:03.674390 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:03.674353    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:50:04.709266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:04.709228    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" path="/var/lib/kubelet/pods/087fda7b-6ff4-4a6e-b49e-f909edcca9c8/volumes"
Oct 30 21:50:04.709730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:04.709715    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a15728c-d24e-4884-b8a5-5be024cc0514" path="/var/lib/kubelet/pods/1a15728c-d24e-4884-b8a5-5be024cc0514/volumes"
Oct 30 21:50:05.552831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.552796    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:50:05.553558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553542    7037 cpu_manager.go:414] "containerMap: removing container" podUID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" containerName="payload-pre-processing"
Oct 30 21:50:05.553628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553558    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" containerName="payload-pre-processing"
Oct 30 21:50:05.553628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553583    7037 cpu_manager.go:414] "containerMap: removing container" podUID="75e85e7e-2402-46f1-9895-bd88d474174f" containerName="main"
Oct 30 21:50:05.553628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553588    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e85e7e-2402-46f1-9895-bd88d474174f" containerName="main"
Oct 30 21:50:05.553628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553619    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1a15728c-d24e-4884-b8a5-5be024cc0514" containerName="payload-processing"
Oct 30 21:50:05.553628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553626    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a15728c-d24e-4884-b8a5-5be024cc0514" containerName="payload-processing"
Oct 30 21:50:05.553807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553735    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="087fda7b-6ff4-4a6e-b49e-f909edcca9c8" containerName="payload-pre-processing"
Oct 30 21:50:05.553807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553746    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e85e7e-2402-46f1-9895-bd88d474174f" containerName="main"
Oct 30 21:50:05.553807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.553757    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a15728c-d24e-4884-b8a5-5be024cc0514" containerName="payload-processing"
Oct 30 21:50:05.558227 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.558207    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.575934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.575910    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e60e16\""
Oct 30 21:50:05.578278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.578251    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e60e16-dockercfg-bt4xx\""
Oct 30 21:50:05.595661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.595307    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:50:05.672440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.672397    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.672649 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.672585    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmtn\" (UniqueName: \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.774652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.774614    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.775124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.774751    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmtn\" (UniqueName: \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.775858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.775825    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.795016 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.794990    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmtn\" (UniqueName: \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\") pod \"payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:05.873379 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:05.873346    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:06.227831 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:06.227801    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856ee13a_a04c_468f_a277_163bf956e0de.slice/crio-4b6beaeb5fe9f5e9a64ddc19aa818bc1c2efa82e445f5353236974f8321266e2 WatchSource:0}: Error finding container 4b6beaeb5fe9f5e9a64ddc19aa818bc1c2efa82e445f5353236974f8321266e2: Status 404 returned error can't find the container with id 4b6beaeb5fe9f5e9a64ddc19aa818bc1c2efa82e445f5353236974f8321266e2
Oct 30 21:50:06.230081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:06.230054    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:50:06.858949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:06.858902    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" event={"ID":"856ee13a-a04c-468f-a277-163bf956e0de","Type":"ContainerStarted","Data":"d296ee51c3df44e82e9a58e93f99fd4ed8ee9c4ca1396b664de78a825a40a388"}
Oct 30 21:50:06.859376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:06.858969    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" event={"ID":"856ee13a-a04c-468f-a277-163bf956e0de","Type":"ContainerStarted","Data":"4b6beaeb5fe9f5e9a64ddc19aa818bc1c2efa82e445f5353236974f8321266e2"}
Oct 30 21:50:06.859376 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:06.859002    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:06.901249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:06.901201    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" podStartSLOduration=1.901187497 podStartE2EDuration="1.901187497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:06.900443534 +0000 UTC m=+2950.311982972" watchObservedRunningTime="2026-10-30 21:50:06.901187497 +0000 UTC m=+2950.312726933"
Oct 30 21:50:08.206295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:08.206243    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing" containerID="cri-o://708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266" gracePeriod=25
Oct 30 21:50:09.340691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:09.340645    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:50:12.492680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.492637    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main" containerID="cri-o://f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4" gracePeriod=45
Oct 30 21:50:12.519978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.519943    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" containerID="cri-o://b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468" gracePeriod=45
Oct 30 21:50:12.523530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.523480    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main" containerID="cri-o://008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55" gracePeriod=45
Oct 30 21:50:12.819837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.819790    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:50:12.862923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.862892    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:50:12.865323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.865294    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:50:12.882944 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.882912    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883171    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883203    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.883273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883227    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883318    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883343    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883372    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\" (UniqueName: \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\") pod \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\" (UID: \"e27f9f09-d48e-4b8b-b857-6d949cf7633c\") "
Oct 30 21:50:12.883586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883448    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.883586 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883525    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.883933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.883910    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.884103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.884081    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.884225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.884106    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.886028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.886001    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.886111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.886033    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "kube-api-access-d82s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.886111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.886036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs" pod "e27f9f09-d48e-4b8b-b857-6d949cf7633c" (UID: "e27f9f09-d48e-4b8b-b857-6d949cf7633c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:12.908070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.908035    7037 generic.go:356] "Generic (PLEG): container finished" podID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerID="f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4" exitCode=0
Oct 30 21:50:12.909610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.908192    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" event={"ID":"512c8a29-d681-4ef8-bc5c-8c21589dad70","Type":"ContainerDied","Data":"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"}
Oct 30 21:50:12.909610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.908266    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz" event={"ID":"512c8a29-d681-4ef8-bc5c-8c21589dad70","Type":"ContainerDied","Data":"53652bdf6e08d894ca6f975bbbd2a4628e329c40189f7aa857b6591f210c4541"}
Oct 30 21:50:12.909610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.908293    7037 scope.go:122] "RemoveContainer" containerID="f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"
Oct 30 21:50:12.909610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.908775    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"
Oct 30 21:50:12.910780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.910212    7037 generic.go:356] "Generic (PLEG): container finished" podID="6c385729-0387-4c4e-af25-12f078af3177" containerID="008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55" exitCode=0
Oct 30 21:50:12.910780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.910541    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"
Oct 30 21:50:12.910780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.910689    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" event={"ID":"6c385729-0387-4c4e-af25-12f078af3177","Type":"ContainerDied","Data":"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"}
Oct 30 21:50:12.910780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.910723    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm" event={"ID":"6c385729-0387-4c4e-af25-12f078af3177","Type":"ContainerDied","Data":"eb00985bd3c5ef08ba5cb24e109adfbeb5ac13d5da939db8bab181cca04afc89"}
Oct 30 21:50:12.914118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.913859    7037 generic.go:356] "Generic (PLEG): container finished" podID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerID="b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468" exitCode=0
Oct 30 21:50:12.914118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.913916    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"
Oct 30 21:50:12.914118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.913941    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" event={"ID":"e27f9f09-d48e-4b8b-b857-6d949cf7633c","Type":"ContainerDied","Data":"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"}
Oct 30 21:50:12.914118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.913978    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" event={"ID":"e27f9f09-d48e-4b8b-b857-6d949cf7633c","Type":"ContainerDied","Data":"2ffdb8c4b2c47fd2ab67b5e36bbc417bbefc09b8129962f0df91bfec76eff608"}
Oct 30 21:50:12.927315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.925844    7037 scope.go:122] "RemoveContainer" containerID="f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"
Oct 30 21:50:12.927315 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:12.926334    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4\": container with ID starting with f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4 not found: ID does not exist" containerID="f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"
Oct 30 21:50:12.927315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.926404    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4"} err="failed to get container status \"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4\": rpc error: code = NotFound desc = could not find container \"f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4\": container with ID starting with f6786a2da8ac4e2aa5114c2e03f40299542d049e8a5ffb6973d519a2fc8afad4 not found: ID does not exist"
Oct 30 21:50:12.927315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.926427    7037 scope.go:122] "RemoveContainer" containerID="008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"
Oct 30 21:50:12.940277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.940252    7037 scope.go:122] "RemoveContainer" containerID="008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"
Oct 30 21:50:12.940738 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:12.940714    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55\": container with ID starting with 008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55 not found: ID does not exist" containerID="008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"
Oct 30 21:50:12.940816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.940745    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55"} err="failed to get container status \"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55\": rpc error: code = NotFound desc = could not find container \"008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55\": container with ID starting with 008a32c289ad7911be5aadc427c335003d70495107cf560447e267e5a3abdb55 not found: ID does not exist"
Oct 30 21:50:12.940816 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.940770    7037 scope.go:122] "RemoveContainer" containerID="b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"
Oct 30 21:50:12.967540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.967513    7037 scope.go:122] "RemoveContainer" containerID="b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"
Oct 30 21:50:12.968482 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:12.968445    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468\": container with ID starting with b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468 not found: ID does not exist" containerID="b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"
Oct 30 21:50:12.968662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.968526    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468"} err="failed to get container status \"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468\": rpc error: code = NotFound desc = could not find container \"b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468\": container with ID starting with b79a9dfcbacb67787bc98fc50449670cba6d76b76a0ff0610dc5ce42bb2fe468 not found: ID does not exist"
Oct 30 21:50:12.985125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985086    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985144    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.985271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985219    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.985271 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985253    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\" (UniqueName: \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985318    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\" (UniqueName: \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.985464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985346    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985413    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\" (UniqueName: \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985467    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.985702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985540    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985607    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.985875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985788    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\") pod \"512c8a29-d681-4ef8-bc5c-8c21589dad70\" (UID: \"512c8a29-d681-4ef8-bc5c-8c21589dad70\") "
Oct 30 21:50:12.985875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985819    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\") pod \"6c385729-0387-4c4e-af25-12f078af3177\" (UID: \"6c385729-0387-4c4e-af25-12f078af3177\") "
Oct 30 21:50:12.986024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.985974    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986043    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986107    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986253    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986434    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986618    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.986721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986655    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.986721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986623    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.986721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986672    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.986721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986703    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e27f9f09-d48e-4b8b-b857-6d949cf7633c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.986721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986718    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.987003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986731    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.987003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986744    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e27f9f09-d48e-4b8b-b857-6d949cf7633c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.987003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.986758    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d82s8\" (UniqueName: \"kubernetes.io/projected/e27f9f09-d48e-4b8b-b857-6d949cf7633c-kube-api-access-d82s8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:12.989888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.989855    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:12.990059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.990038    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "kube-api-access-r5mpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.991077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.991041    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:12.991238 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.991091    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "kube-api-access-s5xn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.992155 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.992133    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm" pod "6c385729-0387-4c4e-af25-12f078af3177" (UID: "6c385729-0387-4c4e-af25-12f078af3177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.994603 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.994581    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm" pod "512c8a29-d681-4ef8-bc5c-8c21589dad70" (UID: "512c8a29-d681-4ef8-bc5c-8c21589dad70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.994996 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.994981    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:50:12.998152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:12.998130    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9"]
Oct 30 21:50:13.088213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088176    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088211    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mpk\" (UniqueName: \"kubernetes.io/projected/512c8a29-d681-4ef8-bc5c-8c21589dad70-kube-api-access-r5mpk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088213 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088227    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xn8\" (UniqueName: \"kubernetes.io/projected/6c385729-0387-4c4e-af25-12f078af3177-kube-api-access-s5xn8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088241    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/512c8a29-d681-4ef8-bc5c-8c21589dad70-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088254    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c385729-0387-4c4e-af25-12f078af3177-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088265    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/512c8a29-d681-4ef8-bc5c-8c21589dad70-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.088538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.088276    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c385729-0387-4c4e-af25-12f078af3177-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:13.259588 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.259558    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:50:13.262554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.262530    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/simulator-w2-2cda44-kserve-5cfdd86d89-dt7pm"]
Oct 30 21:50:13.298458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.298418    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:50:13.301452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.301430    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2cda44/distinct-2-w2-2cda44-kserve-7cc5dc876d-rfvpz"]
Oct 30 21:50:13.741676 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:13.741634    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2cda44/premium-w2-2cda44-kserve-5d4669cd8d-fb2n9" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:50:14.709055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:14.709019    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" path="/var/lib/kubelet/pods/512c8a29-d681-4ef8-bc5c-8c21589dad70/volumes"
Oct 30 21:50:14.709501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:14.709478    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c385729-0387-4c4e-af25-12f078af3177" path="/var/lib/kubelet/pods/6c385729-0387-4c4e-af25-12f078af3177/volumes"
Oct 30 21:50:14.709947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:14.709931    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" path="/var/lib/kubelet/pods/e27f9f09-d48e-4b8b-b857-6d949cf7633c/volumes"
Oct 30 21:50:17.700812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.700777    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:50:17.701536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701520    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main"
Oct 30 21:50:17.701536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701535    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main"
Oct 30 21:50:17.701637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701565    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main"
Oct 30 21:50:17.701637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701570    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main"
Oct 30 21:50:17.701637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701599    7037 cpu_manager.go:414] "containerMap: removing container" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main"
Oct 30 21:50:17.701637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701605    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main"
Oct 30 21:50:17.701778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701722    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27f9f09-d48e-4b8b-b857-6d949cf7633c" containerName="main"
Oct 30 21:50:17.701778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701736    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c385729-0387-4c4e-af25-12f078af3177" containerName="main"
Oct 30 21:50:17.701778 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.701746    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="512c8a29-d681-4ef8-bc5c-8c21589dad70" containerName="main"
Oct 30 21:50:17.707118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.707052    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.728524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.723632    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f9a8ed-serving-cert\""
Oct 30 21:50:17.729125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.729100    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:50:17.854594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.854560    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.854787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.854620    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.854787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.854645    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.854872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.854791    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2ts\" (UniqueName: \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.872073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.872041    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:50:17.957145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.957048    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.957339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.957163    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.957339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.957199    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.957339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.957327    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2ts\" (UniqueName: \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.958573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.958483    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.960848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.960816    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.962032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.962007    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:17.978805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:17.978773    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2ts\" (UniqueName: \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\") pod \"maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:18.024017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.023989    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:18.333466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.333431    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:50:18.333927 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:18.333887    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c29cfd3_fe68_4727_a98b_5408248231c2.slice/crio-916cfc78b27fd3c251eb592e23e4e1da37fe3e4af6e2f12bb8223e345ae74f42 WatchSource:0}: Error finding container 916cfc78b27fd3c251eb592e23e4e1da37fe3e4af6e2f12bb8223e345ae74f42: Status 404 returned error can't find the container with id 916cfc78b27fd3c251eb592e23e4e1da37fe3e4af6e2f12bb8223e345ae74f42
Oct 30 21:50:18.335470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.335447    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:50:18.971638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.971554    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" event={"ID":"7c29cfd3-fe68-4727-a98b-5408248231c2","Type":"ContainerStarted","Data":"bb32b4b58482bbbcb5b62e2a57db647362912eb6a7542f2e275beb94938759b0"}
Oct 30 21:50:18.971638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.971604    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" event={"ID":"7c29cfd3-fe68-4727-a98b-5408248231c2","Type":"ContainerStarted","Data":"916cfc78b27fd3c251eb592e23e4e1da37fe3e4af6e2f12bb8223e345ae74f42"}
Oct 30 21:50:18.972157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:18.971752    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:19.014698 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:19.014637    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" podStartSLOduration=1.672660107 podStartE2EDuration="2.01462288s" totalImagesPullingTime="341.962773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:19.012255513 +0000 UTC m=+2962.423794950" watchObservedRunningTime="2026-10-30 21:50:19.01462288 +0000 UTC m=+2962.426162347"
Oct 30 21:50:19.340653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:19.340609    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:50:24.985369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:24.985338    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:50:26.037713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.037682    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:50:26.042365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.042335    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.061124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.061092    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:50:26.066893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.066860    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.066893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.066897    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.067135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.066993    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.067135 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.067131    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.067256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.067162    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbrb\" (UniqueName: \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.067256 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.067198    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.069652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.069621    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-203209\"/\"default-dockercfg-f7djt\""
Oct 30 21:50:26.069652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.069622    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-203209\"/\"ipp-route-203209-kserve-self-signed-certs\""
Oct 30 21:50:26.070365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.070340    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-203209\"/\"openshift-service-ca.crt\""
Oct 30 21:50:26.070503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.070401    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-203209\"/\"kube-root-ca.crt\""
Oct 30 21:50:26.168185 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168148    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168216    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168351    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168381    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbrb\" (UniqueName: \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168418    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168468    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168746 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168676    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168751    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.168807 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.168753    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.170977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.170946    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.171261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.171242    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.188178 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.188147    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbrb\" (UniqueName: \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\") pod \"ipp-route-203209-kserve-798cf4d8df-zl7c5\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") " pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.362776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.362734    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:26.663038 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:26.662951    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:50:26.668213 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:26.668180    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde6dc9d8_15ef_45fa_8a70_520dfa0fe8e0.slice/crio-74667a6f56b83c4353fd494687a0d68b91ac565215d2b89d27a01bd9e8d19943 WatchSource:0}: Error finding container 74667a6f56b83c4353fd494687a0d68b91ac565215d2b89d27a01bd9e8d19943: Status 404 returned error can't find the container with id 74667a6f56b83c4353fd494687a0d68b91ac565215d2b89d27a01bd9e8d19943
Oct 30 21:50:27.030344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:27.030246    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" event={"ID":"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0","Type":"ContainerStarted","Data":"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"}
Oct 30 21:50:27.030344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:27.030295    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" event={"ID":"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0","Type":"ContainerStarted","Data":"74667a6f56b83c4353fd494687a0d68b91ac565215d2b89d27a01bd9e8d19943"}
Oct 30 21:50:27.072308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:27.072254    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" podStartSLOduration=1.072240176 podStartE2EDuration="1.072240176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:27.069099873 +0000 UTC m=+2970.480639310" watchObservedRunningTime="2026-10-30 21:50:27.072240176 +0000 UTC m=+2970.483779613"
Oct 30 21:50:27.363029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:27.362989    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:28.489325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.489296    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:50:28.598643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.598608    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\" (UniqueName: \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\") pod \"6a226687-037d-47f7-a6e4-2899c18a7ee9\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") "
Oct 30 21:50:28.598853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.598664    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\") pod \"6a226687-037d-47f7-a6e4-2899c18a7ee9\" (UID: \"6a226687-037d-47f7-a6e4-2899c18a7ee9\") "
Oct 30 21:50:28.599167 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.599135    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume" pod "6a226687-037d-47f7-a6e4-2899c18a7ee9" (UID: "6a226687-037d-47f7-a6e4-2899c18a7ee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:28.599367 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.599346    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a226687-037d-47f7-a6e4-2899c18a7ee9-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:28.601118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.601098    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88" pod "6a226687-037d-47f7-a6e4-2899c18a7ee9" (UID: "6a226687-037d-47f7-a6e4-2899c18a7ee9"). InnerVolumeSpecName "kube-api-access-tlm88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:28.700618 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:28.700587    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlm88\" (UniqueName: \"kubernetes.io/projected/6a226687-037d-47f7-a6e4-2899c18a7ee9-kube-api-access-tlm88\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:50:29.050118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.050021    7037 generic.go:356] "Generic (PLEG): container finished" podID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerID="708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266" exitCode=0
Oct 30 21:50:29.050118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.050069    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"
Oct 30 21:50:29.050118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.050108    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" event={"ID":"6a226687-037d-47f7-a6e4-2899c18a7ee9","Type":"ContainerDied","Data":"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"}
Oct 30 21:50:29.050432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.050167    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq" event={"ID":"6a226687-037d-47f7-a6e4-2899c18a7ee9","Type":"ContainerDied","Data":"8725b6b0c4d5b6065778b629cead0af569c8c66f199c7b9c2fc533dff2b1c825"}
Oct 30 21:50:29.050432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.050193    7037 scope.go:122] "RemoveContainer" containerID="708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"
Oct 30 21:50:29.061636 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.061612    7037 scope.go:122] "RemoveContainer" containerID="708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"
Oct 30 21:50:29.061945 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:29.061924    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266\": container with ID starting with 708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266 not found: ID does not exist" containerID="708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"
Oct 30 21:50:29.062020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.061956    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266"} err="failed to get container status \"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266\": rpc error: code = NotFound desc = could not find container \"708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266\": container with ID starting with 708c6027b82eed2dc41874bbf11e4216839a4617d2441443964250f378e7a266 not found: ID does not exist"
Oct 30 21:50:29.105341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.105300    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:50:29.117098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:29.117062    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2cda44-7b68745cf-vp9kq"]
Oct 30 21:50:30.709274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:30.709243    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" path="/var/lib/kubelet/pods/6a226687-037d-47f7-a6e4-2899c18a7ee9/volumes"
Oct 30 21:50:33.673081 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:50:33.673041    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:50:34.062125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.061692    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:50:34.065032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.064690    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing"
Oct 30 21:50:34.065032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.064719    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing"
Oct 30 21:50:34.065032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.064981    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a226687-037d-47f7-a6e4-2899c18a7ee9" containerName="payload-pre-processing"
Oct 30 21:50:34.072096 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.072059    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.090009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.089972    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:50:34.095863 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.095828    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8964ac-dockercfg-8d2bg\""
Oct 30 21:50:34.096023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.095966    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8964ac\""
Oct 30 21:50:34.176896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.176848    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjscc\" (UniqueName: \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.177099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.177008    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.278474 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.278438    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjscc\" (UniqueName: \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.278715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.278566    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.279470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.279448    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.301402 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.301363    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjscc\" (UniqueName: \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\") pod \"payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.407360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.407323    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:34.742708 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:34.742679    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72c4f05_d716_4597_abfd_ec0df8009b0e.slice/crio-b97b564f39e640cab842e00a4eed51c10344f028b87ebd8f0f1f06a4465b8ca6 WatchSource:0}: Error finding container b97b564f39e640cab842e00a4eed51c10344f028b87ebd8f0f1f06a4465b8ca6: Status 404 returned error can't find the container with id b97b564f39e640cab842e00a4eed51c10344f028b87ebd8f0f1f06a4465b8ca6
Oct 30 21:50:34.745084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:34.745057    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:50:35.101467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:35.101426    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" event={"ID":"f72c4f05-d716-4597-abfd-ec0df8009b0e","Type":"ContainerStarted","Data":"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"}
Oct 30 21:50:35.101719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:35.101526    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" event={"ID":"f72c4f05-d716-4597-abfd-ec0df8009b0e","Type":"ContainerStarted","Data":"b97b564f39e640cab842e00a4eed51c10344f028b87ebd8f0f1f06a4465b8ca6"}
Oct 30 21:50:35.101719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:35.101555    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:35.157978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:35.157920    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" podStartSLOduration=1.157905974 podStartE2EDuration="1.157905974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:35.155717318 +0000 UTC m=+2978.567256755" watchObservedRunningTime="2026-10-30 21:50:35.157905974 +0000 UTC m=+2978.569445411"
Oct 30 21:50:36.363181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:36.363087    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:36.376670 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:36.376644    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:37.141031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:37.141000    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:50:46.111337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:46.111306    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:50:55.720907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.720872    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:50:55.725759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.725736    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.745767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.745738    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:50:55.754753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.754728    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e136b8\""
Oct 30 21:50:55.792064 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.792035    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:50:55.796469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.796451    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:55.814554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.814526    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e136b8-dockercfg-n7fl2\""
Oct 30 21:50:55.827576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.817249    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:50:55.887512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.887192    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:55.887512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.887358    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s78j\" (UniqueName: \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.887512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.887395    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872tj\" (UniqueName: \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:55.887830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.887673    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.989194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.989087    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s78j\" (UniqueName: \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.989386 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.989317    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872tj\" (UniqueName: \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:55.989566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.989541    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.989674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.989637    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:55.990722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.990688    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:55.990856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:55.990743    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:56.010034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.009999    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s78j\" (UniqueName: \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\") pod \"payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:56.010229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.009997    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872tj\" (UniqueName: \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\") pod \"payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:56.040700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.040668    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:56.111063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.110767    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:56.355448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.355421    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:50:56.355608 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:56.355587    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711f2153_3954_4347_832f_b9208f039e24.slice/crio-66f2456d36600de1cf728a465b12b5683523768d041d77d5b5097b45adfa69df WatchSource:0}: Error finding container 66f2456d36600de1cf728a465b12b5683523768d041d77d5b5097b45adfa69df: Status 404 returned error can't find the container with id 66f2456d36600de1cf728a465b12b5683523768d041d77d5b5097b45adfa69df
Oct 30 21:50:56.442249 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:50:56.442222    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e2a152_dc2b_48d1_be17_cf25a7f8a1ac.slice/crio-225b51063fa7e830643c8552046b950bca9618d2d19c2c27b65a2ff6d1c619b8 WatchSource:0}: Error finding container 225b51063fa7e830643c8552046b950bca9618d2d19c2c27b65a2ff6d1c619b8: Status 404 returned error can't find the container with id 225b51063fa7e830643c8552046b950bca9618d2d19c2c27b65a2ff6d1c619b8
Oct 30 21:50:56.442731 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:56.442708    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:50:57.289630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.289587    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" event={"ID":"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac","Type":"ContainerStarted","Data":"60e78dea4b8eb52d6b71f538fd6333c0e595cfb4c5679ff8b7f65b242c1631ea"}
Oct 30 21:50:57.290104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.289647    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" event={"ID":"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac","Type":"ContainerStarted","Data":"225b51063fa7e830643c8552046b950bca9618d2d19c2c27b65a2ff6d1c619b8"}
Oct 30 21:50:57.290104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.289825    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:50:57.291772 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.291736    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" event={"ID":"711f2153-3954-4347-832f-b9208f039e24","Type":"ContainerStarted","Data":"ca03b3d7f0d79b28b408da9304f633caf898996bb015fc2de56cd6dd09beaffc"}
Oct 30 21:50:57.291913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.291808    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:50:57.291913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.291834    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" event={"ID":"711f2153-3954-4347-832f-b9208f039e24","Type":"ContainerStarted","Data":"66f2456d36600de1cf728a465b12b5683523768d041d77d5b5097b45adfa69df"}
Oct 30 21:50:57.332977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.332920    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" podStartSLOduration=2.332908324 podStartE2EDuration="2.332908324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:57.330711828 +0000 UTC m=+3000.742251275" watchObservedRunningTime="2026-10-30 21:50:57.332908324 +0000 UTC m=+3000.744447761"
Oct 30 21:50:57.384651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:50:57.384589    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" podStartSLOduration=2.384571225 podStartE2EDuration="2.384571225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:57.379561509 +0000 UTC m=+3000.791100946" watchObservedRunningTime="2026-10-30 21:50:57.384571225 +0000 UTC m=+3000.796110664"
Oct 30 21:51:03.545270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:03.545232    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:51:03.679952 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:51:03.679910    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:51:08.303623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.303590    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:51:08.304049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.304023    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:51:08.546334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.546291    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" podUID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" containerName="payload-pre-processing" containerID="cri-o://0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6" gracePeriod=25
Oct 30 21:51:08.826648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.826621    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:51:08.888581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.888546    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\" (UniqueName: \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\") pod \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") "
Oct 30 21:51:08.888777 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.888643    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\") pod \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\" (UID: \"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7\") "
Oct 30 21:51:08.889083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.889052    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume" pod "a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" (UID: "a1ce0c9d-a48a-4a90-b20b-91d063c8bff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:08.889331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.889309    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:51:08.890964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.890936    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q" pod "a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" (UID: "a1ce0c9d-a48a-4a90-b20b-91d063c8bff7"). InnerVolumeSpecName "kube-api-access-7nq2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:08.991047 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:08.991004    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nq2q\" (UniqueName: \"kubernetes.io/projected/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7-kube-api-access-7nq2q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:51:09.410610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.410569    7037 generic.go:356] "Generic (PLEG): container finished" podID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" containerID="0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6" exitCode=0
Oct 30 21:51:09.411082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.410627    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"
Oct 30 21:51:09.411082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.410647    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" event={"ID":"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7","Type":"ContainerDied","Data":"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"}
Oct 30 21:51:09.411082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.410724    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr" event={"ID":"a1ce0c9d-a48a-4a90-b20b-91d063c8bff7","Type":"ContainerDied","Data":"aae0a0443a813e7281fe983b3803f7aa808318142af0ee1f06ee28bd1812d5d1"}
Oct 30 21:51:09.411082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.410745    7037 scope.go:122] "RemoveContainer" containerID="0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"
Oct 30 21:51:09.422290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.422270    7037 scope.go:122] "RemoveContainer" containerID="0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"
Oct 30 21:51:09.422591 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:51:09.422561    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6\": container with ID starting with 0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6 not found: ID does not exist" containerID="0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"
Oct 30 21:51:09.422715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.422596    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6"} err="failed to get container status \"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6\": rpc error: code = NotFound desc = could not find container \"0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6\": container with ID starting with 0bb2ac52ddc9895af81f2f6a80d01c9f070691d5cd7769e93a88849ff6e3eeb6 not found: ID does not exist"
Oct 30 21:51:09.463404 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.463373    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:51:09.475604 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:09.475575    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9a8ed-676c67c899-v7qrr"]
Oct 30 21:51:10.710020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:10.709981    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" path="/var/lib/kubelet/pods/a1ce0c9d-a48a-4a90-b20b-91d063c8bff7/volumes"
Oct 30 21:51:19.210067 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.210015    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:51:19.210823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.210807    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" containerName="payload-pre-processing"
Oct 30 21:51:19.210892 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.210824    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" containerName="payload-pre-processing"
Oct 30 21:51:19.210990 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.210980    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ce0c9d-a48a-4a90-b20b-91d063c8bff7" containerName="payload-pre-processing"
Oct 30 21:51:19.216464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.216448    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.234069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.234033    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-95692f-openshift-default-dockercfg-q7k6z\""
Oct 30 21:51:19.238415 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238371    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238432    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238480    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238592    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238625    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47lz\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238652    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238688    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238741    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.238853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.238804    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.239926 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.239896    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:51:19.340369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340324    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47lz\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340372    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340408    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340454    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340559    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340731    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340766    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340815    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.340876 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340823    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.341110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340897    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.341110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.340941    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.341214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.341139    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.341214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.341185    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.341330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.341307    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.342942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.342921    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.343276 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.343255    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.360031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.360001    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47lz\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.360152 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.360045    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.534303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.534212    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:19.853343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.853309    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:51:19.859172 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:51:19.859144    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb569ebc_e1ee_4c61_8fab_40f63e6c6be1.slice/crio-9448c6e1203af463766053c1d443c24e8bb848df7d934a5bca5993d8aaa4a67a WatchSource:0}: Error finding container 9448c6e1203af463766053c1d443c24e8bb848df7d934a5bca5993d8aaa4a67a: Status 404 returned error can't find the container with id 9448c6e1203af463766053c1d443c24e8bb848df7d934a5bca5993d8aaa4a67a
Oct 30 21:51:19.862139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.862097    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:19.862221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.862202    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:19.862269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:19.862253    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:20.499004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.498954    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" event={"ID":"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1","Type":"ContainerStarted","Data":"daafcf3b5d02b5a63ffaae8168e14fe3ebf4fbfc67b5e8cf080fc088deaec309"}
Oct 30 21:51:20.499440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.499027    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" event={"ID":"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1","Type":"ContainerStarted","Data":"9448c6e1203af463766053c1d443c24e8bb848df7d934a5bca5993d8aaa4a67a"}
Oct 30 21:51:20.542970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.542930    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:20.543284 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.543242    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:20.543641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.543621    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:20.590340 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:20.590287    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podStartSLOduration=1.5902727300000001 podStartE2EDuration="1.59027273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:20.576002862 +0000 UTC m=+3023.987542299" watchObservedRunningTime="2026-10-30 21:51:20.59027273 +0000 UTC m=+3024.001812158"
Oct 30 21:51:21.548517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.544919    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:21.548517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.544985    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:21.753465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.753425    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:51:21.758576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.758552    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777175    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777238    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777328    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777370    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzd8q\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777440    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777635 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777513    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777637    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777664    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.777750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.777705    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.792943 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.792912    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-908469-openshift-default-dockercfg-bvlc4\""
Oct 30 21:51:21.795011 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.794984    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:51:21.879131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879087    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzd8q\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879164    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879224    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879348    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879386    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879440    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879503    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879541    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879633    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879687    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879706    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879812 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879757    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.879986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.879966    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.880298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.880280    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.881923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.881896    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.882131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.882115    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.898956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.898934    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:21.899062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:21.899024    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzd8q\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:22.075916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.075886    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:22.452083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.450055    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:51:22.452083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.451428    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:22.452083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.451513    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:22.452083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.451548    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:51:22.515575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.515540    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" event={"ID":"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d","Type":"ContainerStarted","Data":"4e6e78e15b0584fdb2cf9ea94ad6d88d2dbc1d25d80ffe370d1d023efad24065"}
Oct 30 21:51:22.550538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.550080    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:22.550538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:22.550133    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:23.545515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:23.538708    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:23.545515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:23.538768    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:23.573517 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:23.572475    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" event={"ID":"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d","Type":"ContainerStarted","Data":"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"}
Oct 30 21:51:23.648686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:23.648628    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podStartSLOduration=2.648605647 podStartE2EDuration="2.648605647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.632015445 +0000 UTC m=+3027.043554883" watchObservedRunningTime="2026-10-30 21:51:23.648605647 +0000 UTC m=+3027.060145085"
Oct 30 21:51:24.077060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:24.076894    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:24.078463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:24.078368    7037 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:51:24.078463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:24.078417    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 21:51:24.544575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:24.542012    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:24.544575 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:24.542079    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:25.076936 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:25.076893    7037 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:51:25.077455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:25.076952    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 21:51:25.537576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:25.535069    7037 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:51:25.537576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:25.535132    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:51:26.076637 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:26.076600    7037 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:51:26.076831 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:26.076653    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 21:51:26.844273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:26.844236    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:26.846312 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:26.846290    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:26.848709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:26.848687    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:51:27.077039 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:27.077004    7037 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:51:27.077218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:27.077061    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 21:51:28.077243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:28.077205    7037 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:51:28.077716 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:28.077270    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Oct 30 21:51:29.321298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:29.321256    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:29.322797 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:29.322754    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:29.324228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:51:29.324150    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:51:33.674033 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:51:33.673996    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:52:03.609006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.608962    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:52:03.625295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.625264    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.632634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.632601    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:52:03.645722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.645688    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be3ada\""
Oct 30 21:52:03.687363 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:52:03.687248    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:52:03.783721 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.783668    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjtq\" (UniqueName: \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.784082 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.784060    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.886648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.886555    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.886817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.886755    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjtq\" (UniqueName: \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.887334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.887310    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.906200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.906166    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjtq\" (UniqueName: \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\") pod \"payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:03.948280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:03.948243    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:04.250909 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:52:04.250876    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5080b2dd_7157_453b_9323_90cbbb50e6ac.slice/crio-11bd8e260b4d12997ede6fdec3872666a7720427c7a84e48b7eb379c67dbcd80 WatchSource:0}: Error finding container 11bd8e260b4d12997ede6fdec3872666a7720427c7a84e48b7eb379c67dbcd80: Status 404 returned error can't find the container with id 11bd8e260b4d12997ede6fdec3872666a7720427c7a84e48b7eb379c67dbcd80
Oct 30 21:52:04.252717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:04.252694    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:52:04.894448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:04.894411    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" event={"ID":"5080b2dd-7157-453b-9323-90cbbb50e6ac","Type":"ContainerStarted","Data":"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"}
Oct 30 21:52:04.894886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:04.894479    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:04.894886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:04.894518    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" event={"ID":"5080b2dd-7157-453b-9323-90cbbb50e6ac","Type":"ContainerStarted","Data":"11bd8e260b4d12997ede6fdec3872666a7720427c7a84e48b7eb379c67dbcd80"}
Oct 30 21:52:04.935817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:04.935760    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" podStartSLOduration=1.935745484 podStartE2EDuration="1.935745484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:04.934743571 +0000 UTC m=+3068.346283010" watchObservedRunningTime="2026-10-30 21:52:04.935745484 +0000 UTC m=+3068.347284920"
Oct 30 21:52:15.905954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:15.905924    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:52:25.927934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:25.927886    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:52:25.963288 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:25.963253    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:52:27.274070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.274024    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:52:27.285013 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.284977    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.296343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.296308    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:52:27.339815 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.339761    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f8137c\""
Oct 30 21:52:27.340086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.339845    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f8137c-dockercfg-mw9qf\""
Oct 30 21:52:27.441310 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.441272    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rl9x\" (UniqueName: \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.441534 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.441482    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.543045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.542955    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.543231 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.543117    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rl9x\" (UniqueName: \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.543767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.543740    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.563742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.563711    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rl9x\" (UniqueName: \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\") pod \"payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.615540 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.615508    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:27.915921 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:52:27.915880    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1168bc_7bd1_42bc_8c74_b4e700b46377.slice/crio-25cd88fde7c6e2f674e394b558ebf5b8f8e9f098044d7317722898adbfa8f8fc WatchSource:0}: Error finding container 25cd88fde7c6e2f674e394b558ebf5b8f8e9f098044d7317722898adbfa8f8fc: Status 404 returned error can't find the container with id 25cd88fde7c6e2f674e394b558ebf5b8f8e9f098044d7317722898adbfa8f8fc
Oct 30 21:52:27.916350 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:27.916329    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:52:28.066092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:28.066049    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" event={"ID":"5d1168bc-7bd1-42bc-8c74-b4e700b46377","Type":"ContainerStarted","Data":"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"}
Oct 30 21:52:28.066289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:28.066110    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" event={"ID":"5d1168bc-7bd1-42bc-8c74-b4e700b46377","Type":"ContainerStarted","Data":"25cd88fde7c6e2f674e394b558ebf5b8f8e9f098044d7317722898adbfa8f8fc"}
Oct 30 21:52:28.066289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:28.066138    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:28.110817 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:28.110758    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" podStartSLOduration=1.110739393 podStartE2EDuration="1.110739393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:28.107806815 +0000 UTC m=+3091.519346253" watchObservedRunningTime="2026-10-30 21:52:28.110739393 +0000 UTC m=+3091.522278833"
Oct 30 21:52:30.928193 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:30.928150    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" podUID="711f2153-3954-4347-832f-b9208f039e24" containerName="payload-processing" containerID="cri-o://ca03b3d7f0d79b28b408da9304f633caf898996bb015fc2de56cd6dd09beaffc" gracePeriod=25
Oct 30 21:52:30.963887 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:30.963790    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" podUID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" containerName="payload-pre-processing" containerID="cri-o://60e78dea4b8eb52d6b71f538fd6333c0e595cfb4c5679ff8b7f65b242c1631ea" gracePeriod=25
Oct 30 21:52:31.090106 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.090074    7037 generic.go:356] "Generic (PLEG): container finished" podID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" containerID="60e78dea4b8eb52d6b71f538fd6333c0e595cfb4c5679ff8b7f65b242c1631ea" exitCode=0
Oct 30 21:52:31.090264 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.090150    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" event={"ID":"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac","Type":"ContainerDied","Data":"60e78dea4b8eb52d6b71f538fd6333c0e595cfb4c5679ff8b7f65b242c1631ea"}
Oct 30 21:52:31.092373 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.092341    7037 generic.go:356] "Generic (PLEG): container finished" podID="711f2153-3954-4347-832f-b9208f039e24" containerID="ca03b3d7f0d79b28b408da9304f633caf898996bb015fc2de56cd6dd09beaffc" exitCode=0
Oct 30 21:52:31.092531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.092412    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" event={"ID":"711f2153-3954-4347-832f-b9208f039e24","Type":"ContainerDied","Data":"ca03b3d7f0d79b28b408da9304f633caf898996bb015fc2de56cd6dd09beaffc"}
Oct 30 21:52:31.273046 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.273013    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:52:31.307617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.307592    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:52:31.396779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.396743    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\") pod \"711f2153-3954-4347-832f-b9208f039e24\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") "
Oct 30 21:52:31.396962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.396838    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\" (UniqueName: \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\") pod \"711f2153-3954-4347-832f-b9208f039e24\" (UID: \"711f2153-3954-4347-832f-b9208f039e24\") "
Oct 30 21:52:31.397204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.397177    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume" pod "711f2153-3954-4347-832f-b9208f039e24" (UID: "711f2153-3954-4347-832f-b9208f039e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:31.397295 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.397281    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711f2153-3954-4347-832f-b9208f039e24-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:31.399320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.399296    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j" pod "711f2153-3954-4347-832f-b9208f039e24" (UID: "711f2153-3954-4347-832f-b9208f039e24"). InnerVolumeSpecName "kube-api-access-8s78j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:31.498146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.498050    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\") pod \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") "
Oct 30 21:52:31.498328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.498309    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\" (UniqueName: \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\") pod \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\" (UID: \"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac\") "
Oct 30 21:52:31.498441 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.498415    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume" pod "31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" (UID: "31e2a152-dc2b-48d1-be17-cf25a7f8a1ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:31.498956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.498933    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s78j\" (UniqueName: \"kubernetes.io/projected/711f2153-3954-4347-832f-b9208f039e24-kube-api-access-8s78j\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:31.499057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.498958    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:31.500509 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.500475    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj" pod "31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" (UID: "31e2a152-dc2b-48d1-be17-cf25a7f8a1ac"). InnerVolumeSpecName "kube-api-access-872tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:31.600503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:31.600448    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872tj\" (UniqueName: \"kubernetes.io/projected/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac-kube-api-access-872tj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:32.101392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.101361    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"
Oct 30 21:52:32.101902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.101370    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf" event={"ID":"31e2a152-dc2b-48d1-be17-cf25a7f8a1ac","Type":"ContainerDied","Data":"225b51063fa7e830643c8552046b950bca9618d2d19c2c27b65a2ff6d1c619b8"}
Oct 30 21:52:32.101902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.101522    7037 scope.go:122] "RemoveContainer" containerID="60e78dea4b8eb52d6b71f538fd6333c0e595cfb4c5679ff8b7f65b242c1631ea"
Oct 30 21:52:32.103073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.103056    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"
Oct 30 21:52:32.103073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.103060    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9" event={"ID":"711f2153-3954-4347-832f-b9208f039e24","Type":"ContainerDied","Data":"66f2456d36600de1cf728a465b12b5683523768d041d77d5b5097b45adfa69df"}
Oct 30 21:52:32.113436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.113412    7037 scope.go:122] "RemoveContainer" containerID="ca03b3d7f0d79b28b408da9304f633caf898996bb015fc2de56cd6dd09beaffc"
Oct 30 21:52:32.157942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.157906    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:52:32.168452 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.168425    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e136b8-c8c99b868-b7jb9"]
Oct 30 21:52:32.196456 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.196423    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:52:32.199539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.199516    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e136b8-76578f5667-5tsqf"]
Oct 30 21:52:32.709547 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.709514    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" path="/var/lib/kubelet/pods/31e2a152-dc2b-48d1-be17-cf25a7f8a1ac/volumes"
Oct 30 21:52:32.709994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:32.709979    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711f2153-3954-4347-832f-b9208f039e24" path="/var/lib/kubelet/pods/711f2153-3954-4347-832f-b9208f039e24/volumes"
Oct 30 21:52:33.673301 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:52:33.673264    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:52:39.077175 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:39.077145    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:52:45.676368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.676332    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:52:45.677124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677106    7037 cpu_manager.go:414] "containerMap: removing container" podUID="711f2153-3954-4347-832f-b9208f039e24" containerName="payload-processing"
Oct 30 21:52:45.677170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677125    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711f2153-3954-4347-832f-b9208f039e24" containerName="payload-processing"
Oct 30 21:52:45.677170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677145    7037 cpu_manager.go:414] "containerMap: removing container" podUID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" containerName="payload-pre-processing"
Oct 30 21:52:45.677170 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677150    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" containerName="payload-pre-processing"
Oct 30 21:52:45.677318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677308    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="711f2153-3954-4347-832f-b9208f039e24" containerName="payload-processing"
Oct 30 21:52:45.677359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.677322    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e2a152-dc2b-48d1-be17-cf25a7f8a1ac" containerName="payload-pre-processing"
Oct 30 21:52:45.682865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.682843    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.703800 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.703765    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:52:45.710571 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.710531    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-be3ada-serving-cert\""
Oct 30 21:52:45.791920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.791881    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.792127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.792020    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.792397 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.792366    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxczx\" (UniqueName: \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.792565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.792547    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.894246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.894206    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.894435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.894280    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.894435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.894327    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxczx\" (UniqueName: \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.894435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.894392    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.895085 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.895058    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.896855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.896833    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.897045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.896926    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:45.915781 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:45.915750    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxczx\" (UniqueName: \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\") pod \"maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:46.000218 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:46.000124    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:46.302124 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:46.302090    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:52:46.302789 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:52:46.302758    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7766e_85f0_47ef_a75f_dd2e064114db.slice/crio-88dc9e501208854763503204e231e457f8daf8744d8b80eac3ca38a4d799469d WatchSource:0}: Error finding container 88dc9e501208854763503204e231e457f8daf8744d8b80eac3ca38a4d799469d: Status 404 returned error can't find the container with id 88dc9e501208854763503204e231e457f8daf8744d8b80eac3ca38a4d799469d
Oct 30 21:52:47.218750 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:47.218702    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" event={"ID":"c9c7766e-85f0-47ef-a75f-dd2e064114db","Type":"ContainerStarted","Data":"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"}
Oct 30 21:52:47.219232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:47.218798    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:47.219232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:47.218816    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" event={"ID":"c9c7766e-85f0-47ef-a75f-dd2e064114db","Type":"ContainerStarted","Data":"88dc9e501208854763503204e231e457f8daf8744d8b80eac3ca38a4d799469d"}
Oct 30 21:52:47.261556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:47.261500    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" podStartSLOduration=1.959523605 podStartE2EDuration="2.261471745s" totalImagesPullingTime="301.94814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:47.259947146 +0000 UTC m=+3110.671486576" watchObservedRunningTime="2026-10-30 21:52:47.261471745 +0000 UTC m=+3110.673011182"
Oct 30 21:52:52.147515 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.147444    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:52:52.147977 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.147704    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" podUID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" containerName="maas-api" containerID="cri-o://a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4" gracePeriod=30
Oct 30 21:52:52.449928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.449901    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:52:52.593118 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.593085    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\") pod \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") "
Oct 30 21:52:52.593298 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.593247    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\") pod \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") "
Oct 30 21:52:52.593344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.593313    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\" (UniqueName: \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\") pod \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") "
Oct 30 21:52:52.593554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.593516    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\") pod \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\" (UID: \"da6d6e3a-0722-4b2b-8272-2d52086cd8ab\") "
Oct 30 21:52:52.593954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.593926    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca" pod "da6d6e3a-0722-4b2b-8272-2d52086cd8ab" (UID: "da6d6e3a-0722-4b2b-8272-2d52086cd8ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:52.594201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.594180    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:52.595464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.595439    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p" pod "da6d6e3a-0722-4b2b-8272-2d52086cd8ab" (UID: "da6d6e3a-0722-4b2b-8272-2d52086cd8ab"). InnerVolumeSpecName "kube-api-access-6vd4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:52.595576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.595548    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls" pod "da6d6e3a-0722-4b2b-8272-2d52086cd8ab" (UID: "da6d6e3a-0722-4b2b-8272-2d52086cd8ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:52.595576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.595565    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs" pod "da6d6e3a-0722-4b2b-8272-2d52086cd8ab" (UID: "da6d6e3a-0722-4b2b-8272-2d52086cd8ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:52.698858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.698757    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:52.698858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.698799    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:52.698858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:52.698814    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vd4p\" (UniqueName: \"kubernetes.io/projected/da6d6e3a-0722-4b2b-8272-2d52086cd8ab-kube-api-access-6vd4p\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:52:53.233257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.233229    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:52:53.270752 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.270717    7037 generic.go:356] "Generic (PLEG): container finished" podID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" containerID="a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4" exitCode=0
Oct 30 21:52:53.270956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.270766    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"
Oct 30 21:52:53.270956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.270803    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" event={"ID":"da6d6e3a-0722-4b2b-8272-2d52086cd8ab","Type":"ContainerDied","Data":"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"}
Oct 30 21:52:53.270956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.270864    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn" event={"ID":"da6d6e3a-0722-4b2b-8272-2d52086cd8ab","Type":"ContainerDied","Data":"55d98bf96d1f03e2a7be1d7fda08db4a205d4a3cd296e0d0ebff15b918407867"}
Oct 30 21:52:53.270956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.270895    7037 scope.go:122] "RemoveContainer" containerID="a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"
Oct 30 21:52:53.284539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.284516    7037 scope.go:122] "RemoveContainer" containerID="a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"
Oct 30 21:52:53.284841 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:52:53.284820    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4\": container with ID starting with a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4 not found: ID does not exist" containerID="a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"
Oct 30 21:52:53.284925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.284854    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4"} err="failed to get container status \"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4\": rpc error: code = NotFound desc = could not find container \"a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4\": container with ID starting with a2bf1f8f27691f1f27e5eb1eb16af4d25fd844faa15da9af9de09ad5f33086a4 not found: ID does not exist"
Oct 30 21:52:53.318253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.318221    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:52:53.324763 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:53.324735    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1d376-d948489fd-vw5hn"]
Oct 30 21:52:54.709529 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:52:54.709481    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" path="/var/lib/kubelet/pods/da6d6e3a-0722-4b2b-8272-2d52086cd8ab/volumes"
Oct 30 21:53:03.674159 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:53:03.674113    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:53:10.080374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.080333    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:53:10.081407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.081380    7037 cpu_manager.go:414] "containerMap: removing container" podUID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" containerName="maas-api"
Oct 30 21:53:10.081407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.081408    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" containerName="maas-api"
Oct 30 21:53:10.081667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.081652    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6d6e3a-0722-4b2b-8272-2d52086cd8ab" containerName="maas-api"
Oct 30 21:53:10.087007 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.086982    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.106844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.106813    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f8137c-serving-cert\""
Oct 30 21:53:10.110694 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.110652    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:53:10.143136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.143104    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46xj\" (UniqueName: \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.143357 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.143169    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.143445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.143369    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.143601 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.143581    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.245645 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.245593    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.246432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.246402    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.246693 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:53:10.246570    7037 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-serving-cert: secret "maas-api-e2e-ipp-b-f8137c-serving-cert" not found
Oct 30 21:53:10.246773 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:53:10.246746    7037 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls podName:e0e3def0-0bfc-4501-95c7-bafc91844aef nodeName:}" failed. No retries permitted until 2026-10-30 21:53:10.74672853 +0000 UTC m=+3134.158267945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls") pod "maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" (UID: "e0e3def0-0bfc-4501-95c7-bafc91844aef") : secret "maas-api-e2e-ipp-b-f8137c-serving-cert" not found
Oct 30 21:53:10.246893 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.246867    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46xj\" (UniqueName: \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.246993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.246976    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.247074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.247050    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.250033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.249988    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.269290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.269253    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46xj\" (UniqueName: \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.753784 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.753748    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:10.756287 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:10.756248    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:11.004258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:11.004173    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:11.315961 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:53:11.315929    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e3def0_0bfc_4501_95c7_bafc91844aef.slice/crio-310cd6dae521138e7354b3f81a91a7e971a7935712c2eb3e7bb4a369751994a4 WatchSource:0}: Error finding container 310cd6dae521138e7354b3f81a91a7e971a7935712c2eb3e7bb4a369751994a4: Status 404 returned error can't find the container with id 310cd6dae521138e7354b3f81a91a7e971a7935712c2eb3e7bb4a369751994a4
Oct 30 21:53:11.317459 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:11.317435    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:53:11.423437 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:11.423379    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" event={"ID":"e0e3def0-0bfc-4501-95c7-bafc91844aef","Type":"ContainerStarted","Data":"310cd6dae521138e7354b3f81a91a7e971a7935712c2eb3e7bb4a369751994a4"}
Oct 30 21:53:12.432810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:12.432772    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" event={"ID":"e0e3def0-0bfc-4501-95c7-bafc91844aef","Type":"ContainerStarted","Data":"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"}
Oct 30 21:53:12.433239 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:12.432869    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:12.476643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:12.476587    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" podStartSLOduration=2.122823642 podStartE2EDuration="2.476570723s" totalImagesPullingTime="353.747081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:12.473171264 +0000 UTC m=+3135.884710700" watchObservedRunningTime="2026-10-30 21:53:12.476570723 +0000 UTC m=+3135.888110166"
Oct 30 21:53:18.444849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:18.444814    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:53:33.673573 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:53:33.673534    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:53:33.771503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:33.771463    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:53:38.771989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:38.771903    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" podUID="5080b2dd-7157-453b-9323-90cbbb50e6ac" containerName="payload-pre-processing" containerID="cri-o://317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8" gracePeriod=25
Oct 30 21:53:39.057079 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.057046    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:53:39.153065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.153026    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\") pod \"5080b2dd-7157-453b-9323-90cbbb50e6ac\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") "
Oct 30 21:53:39.153277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.153125    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\" (UniqueName: \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\") pod \"5080b2dd-7157-453b-9323-90cbbb50e6ac\" (UID: \"5080b2dd-7157-453b-9323-90cbbb50e6ac\") "
Oct 30 21:53:39.153476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.153453    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume" pod "5080b2dd-7157-453b-9323-90cbbb50e6ac" (UID: "5080b2dd-7157-453b-9323-90cbbb50e6ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:39.153753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.153732    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5080b2dd-7157-453b-9323-90cbbb50e6ac-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:39.155243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.155222    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq" pod "5080b2dd-7157-453b-9323-90cbbb50e6ac" (UID: "5080b2dd-7157-453b-9323-90cbbb50e6ac"). InnerVolumeSpecName "kube-api-access-pqjtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:39.254767 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.254726    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjtq\" (UniqueName: \"kubernetes.io/projected/5080b2dd-7157-453b-9323-90cbbb50e6ac-kube-api-access-pqjtq\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:39.635263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.635227    7037 generic.go:356] "Generic (PLEG): container finished" podID="5080b2dd-7157-453b-9323-90cbbb50e6ac" containerID="317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8" exitCode=0
Oct 30 21:53:39.635448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.635274    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"
Oct 30 21:53:39.635448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.635310    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" event={"ID":"5080b2dd-7157-453b-9323-90cbbb50e6ac","Type":"ContainerDied","Data":"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"}
Oct 30 21:53:39.635448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.635366    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p" event={"ID":"5080b2dd-7157-453b-9323-90cbbb50e6ac","Type":"ContainerDied","Data":"11bd8e260b4d12997ede6fdec3872666a7720427c7a84e48b7eb379c67dbcd80"}
Oct 30 21:53:39.635448 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.635390    7037 scope.go:122] "RemoveContainer" containerID="317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"
Oct 30 21:53:39.647409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.647385    7037 scope.go:122] "RemoveContainer" containerID="317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"
Oct 30 21:53:39.647745 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:53:39.647727    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8\": container with ID starting with 317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8 not found: ID does not exist" containerID="317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"
Oct 30 21:53:39.647804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.647752    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8"} err="failed to get container status \"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8\": rpc error: code = NotFound desc = could not find container \"317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8\": container with ID starting with 317e193897ad5b3c21d6b2b08abb0ad48f13ca9f5edcb2c7abd892c7cb13e8f8 not found: ID does not exist"
Oct 30 21:53:39.692149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.692116    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:53:39.704755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:39.704723    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be3ada-8ddf8c555-6sn9p"]
Oct 30 21:53:40.708911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:40.708881    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5080b2dd-7157-453b-9323-90cbbb50e6ac" path="/var/lib/kubelet/pods/5080b2dd-7157-453b-9323-90cbbb50e6ac/volumes"
Oct 30 21:53:55.500285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.500253    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:53:55.500764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.500466    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" podUID="7c29cfd3-fe68-4727-a98b-5408248231c2" containerName="maas-api" containerID="cri-o://bb32b4b58482bbbcb5b62e2a57db647362912eb6a7542f2e275beb94938759b0" gracePeriod=30
Oct 30 21:53:55.763408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.763322    7037 generic.go:356] "Generic (PLEG): container finished" podID="7c29cfd3-fe68-4727-a98b-5408248231c2" containerID="bb32b4b58482bbbcb5b62e2a57db647362912eb6a7542f2e275beb94938759b0" exitCode=0
Oct 30 21:53:55.763408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.763374    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" event={"ID":"7c29cfd3-fe68-4727-a98b-5408248231c2","Type":"ContainerDied","Data":"bb32b4b58482bbbcb5b62e2a57db647362912eb6a7542f2e275beb94938759b0"}
Oct 30 21:53:55.824685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.824657    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:53:55.899507 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.899457    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\") pod \"7c29cfd3-fe68-4727-a98b-5408248231c2\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") "
Oct 30 21:53:55.899702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.899634    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\") pod \"7c29cfd3-fe68-4727-a98b-5408248231c2\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") "
Oct 30 21:53:55.899907 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.899884    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\" (UniqueName: \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\") pod \"7c29cfd3-fe68-4727-a98b-5408248231c2\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") "
Oct 30 21:53:55.899978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.899930    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\") pod \"7c29cfd3-fe68-4727-a98b-5408248231c2\" (UID: \"7c29cfd3-fe68-4727-a98b-5408248231c2\") "
Oct 30 21:53:55.900032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.900011    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca" pod "7c29cfd3-fe68-4727-a98b-5408248231c2" (UID: "7c29cfd3-fe68-4727-a98b-5408248231c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:55.900587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.900563    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c29cfd3-fe68-4727-a98b-5408248231c2-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:55.901964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.901935    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls" pod "7c29cfd3-fe68-4727-a98b-5408248231c2" (UID: "7c29cfd3-fe68-4727-a98b-5408248231c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:55.902169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.902149    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts" pod "7c29cfd3-fe68-4727-a98b-5408248231c2" (UID: "7c29cfd3-fe68-4727-a98b-5408248231c2"). InnerVolumeSpecName "kube-api-access-qz2ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:55.902351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:55.902327    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs" pod "7c29cfd3-fe68-4727-a98b-5408248231c2" (UID: "7c29cfd3-fe68-4727-a98b-5408248231c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:56.001700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.001667    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:56.001700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.001695    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2ts\" (UniqueName: \"kubernetes.io/projected/7c29cfd3-fe68-4727-a98b-5408248231c2-kube-api-access-qz2ts\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:56.001700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.001706    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c29cfd3-fe68-4727-a98b-5408248231c2-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:53:56.772646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.772616    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"
Oct 30 21:53:56.773115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.772650    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5" event={"ID":"7c29cfd3-fe68-4727-a98b-5408248231c2","Type":"ContainerDied","Data":"916cfc78b27fd3c251eb592e23e4e1da37fe3e4af6e2f12bb8223e345ae74f42"}
Oct 30 21:53:56.773115 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.772712    7037 scope.go:122] "RemoveContainer" containerID="bb32b4b58482bbbcb5b62e2a57db647362912eb6a7542f2e275beb94938759b0"
Oct 30 21:53:56.828857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.828823    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:53:56.839653 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:56.839623    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9a8ed-76445c698b-jgtx5"]
Oct 30 21:53:57.445232 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:57.445202    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:53:58.712066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:53:58.712034    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c29cfd3-fe68-4727-a98b-5408248231c2" path="/var/lib/kubelet/pods/7c29cfd3-fe68-4727-a98b-5408248231c2/volumes"
Oct 30 21:54:02.396296 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.396261    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:54:02.396909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.396596    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" podUID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" containerName="maas-api" containerID="cri-o://792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050" gracePeriod=30
Oct 30 21:54:02.445621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.445574    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" podUID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" containerName="payload-pre-processing" containerID="cri-o://8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9" gracePeriod=25
Oct 30 21:54:02.786463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.786438    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:54:02.790025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.790003    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:54:02.806504 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806461    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\") pod \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") "
Oct 30 21:54:02.806642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806582    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\") pod \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") "
Oct 30 21:54:02.806715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806696    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\") pod \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") "
Oct 30 21:54:02.806773 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806753    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\") pod \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") "
Oct 30 21:54:02.806868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806854    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\" (UniqueName: \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\") pod \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\" (UID: \"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0\") "
Oct 30 21:54:02.806949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.806919    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\" (UniqueName: \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\") pod \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\" (UID: \"5d1168bc-7bd1-42bc-8c74-b4e700b46377\") "
Oct 30 21:54:02.807123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.807095    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca" pod "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" (UID: "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:02.807904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.807582    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume" pod "5d1168bc-7bd1-42bc-8c74-b4e700b46377" (UID: "5d1168bc-7bd1-42bc-8c74-b4e700b46377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:02.807904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.807690    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1168bc-7bd1-42bc-8c74-b4e700b46377-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.807904 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.807711    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.809017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.808995    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls" pod "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" (UID: "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:02.809281 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.809264    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs" pod "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" (UID: "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:02.809740 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.809716    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x" pod "5d1168bc-7bd1-42bc-8c74-b4e700b46377" (UID: "5d1168bc-7bd1-42bc-8c74-b4e700b46377"). InnerVolumeSpecName "kube-api-access-8rl9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:02.810405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.810385    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn" pod "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" (UID: "17c568bd-dbd0-4164-9e2c-1b36b1cd90d0"). InnerVolumeSpecName "kube-api-access-kp9sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:02.823578 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.823554    7037 generic.go:356] "Generic (PLEG): container finished" podID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" containerID="792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050" exitCode=0
Oct 30 21:54:02.823718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.823585    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" event={"ID":"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0","Type":"ContainerDied","Data":"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"}
Oct 30 21:54:02.823718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.823611    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"
Oct 30 21:54:02.823718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.823661    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn" event={"ID":"17c568bd-dbd0-4164-9e2c-1b36b1cd90d0","Type":"ContainerDied","Data":"eab2d488c4608f5aebb3a172b6bc775e6c6cfd7e9f43b59189cae56a167100da"}
Oct 30 21:54:02.823718 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.823691    7037 scope.go:122] "RemoveContainer" containerID="792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"
Oct 30 21:54:02.826967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.826699    7037 generic.go:356] "Generic (PLEG): container finished" podID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" containerID="8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9" exitCode=0
Oct 30 21:54:02.826967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.826775    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" event={"ID":"5d1168bc-7bd1-42bc-8c74-b4e700b46377","Type":"ContainerDied","Data":"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"}
Oct 30 21:54:02.826967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.826827    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt" event={"ID":"5d1168bc-7bd1-42bc-8c74-b4e700b46377","Type":"ContainerDied","Data":"25cd88fde7c6e2f674e394b558ebf5b8f8e9f098044d7317722898adbfa8f8fc"}
Oct 30 21:54:02.826967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.826869    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"
Oct 30 21:54:02.843734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.843711    7037 scope.go:122] "RemoveContainer" containerID="792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"
Oct 30 21:54:02.844096 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:02.844065    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050\": container with ID starting with 792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050 not found: ID does not exist" containerID="792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"
Oct 30 21:54:02.844214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.844107    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050"} err="failed to get container status \"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050\": rpc error: code = NotFound desc = could not find container \"792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050\": container with ID starting with 792f4fad0d2a5337ba4f04c19df60abf28026cfb42fc0310541aee82e8586050 not found: ID does not exist"
Oct 30 21:54:02.844214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.844130    7037 scope.go:122] "RemoveContainer" containerID="8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"
Oct 30 21:54:02.858457 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.858432    7037 scope.go:122] "RemoveContainer" containerID="8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"
Oct 30 21:54:02.858947 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:02.858909    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9\": container with ID starting with 8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9 not found: ID does not exist" containerID="8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"
Oct 30 21:54:02.858947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.858953    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9"} err="failed to get container status \"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9\": rpc error: code = NotFound desc = could not find container \"8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9\": container with ID starting with 8fb210e2041a9444d4164c4cef4aa542edfcf46bbab51484cd6c0eed25c596c9 not found: ID does not exist"
Oct 30 21:54:02.926424 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.924374    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9sn\" (UniqueName: \"kubernetes.io/projected/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-kube-api-access-kp9sn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.926674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.926649    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rl9x\" (UniqueName: \"kubernetes.io/projected/5d1168bc-7bd1-42bc-8c74-b4e700b46377-kube-api-access-8rl9x\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.926792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.926779    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.926875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.926866    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.949942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.949897    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:54:02.954341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.954306    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-203209-d56fc44b8-npvsn"]
Oct 30 21:54:02.984255 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.984208    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:54:02.987617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:02.987593    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f8137c-cc58f54ff-dlqvt"]
Oct 30 21:54:03.674890 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:03.674852    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:54:04.710680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:04.710645    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" path="/var/lib/kubelet/pods/17c568bd-dbd0-4164-9e2c-1b36b1cd90d0/volumes"
Oct 30 21:54:04.711360 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:04.711336    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" path="/var/lib/kubelet/pods/5d1168bc-7bd1-42bc-8c74-b4e700b46377/volumes"
Oct 30 21:54:07.755461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.755284    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:54:07.756221 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.755617    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" podUID="d1178672-53f6-4fd6-a680-de16cb780487" containerName="maas-api" containerID="cri-o://7732909fce5a79a11f72911c85befd433a74011455cc2412e7a9e9a43822fc37" gracePeriod=30
Oct 30 21:54:07.871855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.871824    7037 generic.go:356] "Generic (PLEG): container finished" podID="d1178672-53f6-4fd6-a680-de16cb780487" containerID="7732909fce5a79a11f72911c85befd433a74011455cc2412e7a9e9a43822fc37" exitCode=0
Oct 30 21:54:07.872062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.871900    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" event={"ID":"d1178672-53f6-4fd6-a680-de16cb780487","Type":"ContainerDied","Data":"7732909fce5a79a11f72911c85befd433a74011455cc2412e7a9e9a43822fc37"}
Oct 30 21:54:07.946428 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.946382    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:54:07.946730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:07.946700    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" podUID="f72c4f05-d716-4597-abfd-ec0df8009b0e" containerName="payload-pre-processing" containerID="cri-o://bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968" gracePeriod=30
Oct 30 21:54:08.119965 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.119931    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:54:08.195261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.195229    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\") pod \"d1178672-53f6-4fd6-a680-de16cb780487\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") "
Oct 30 21:54:08.195261 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.195272    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\") pod \"d1178672-53f6-4fd6-a680-de16cb780487\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") "
Oct 30 21:54:08.195557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.195387    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\" (UniqueName: \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\") pod \"d1178672-53f6-4fd6-a680-de16cb780487\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") "
Oct 30 21:54:08.195630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.195561    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\") pod \"d1178672-53f6-4fd6-a680-de16cb780487\" (UID: \"d1178672-53f6-4fd6-a680-de16cb780487\") "
Oct 30 21:54:08.196315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.196089    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca" pod "d1178672-53f6-4fd6-a680-de16cb780487" (UID: "d1178672-53f6-4fd6-a680-de16cb780487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:08.196315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.196288    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1178672-53f6-4fd6-a680-de16cb780487-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.198471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.198441    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs" pod "d1178672-53f6-4fd6-a680-de16cb780487" (UID: "d1178672-53f6-4fd6-a680-de16cb780487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:08.198631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.198607    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct" pod "d1178672-53f6-4fd6-a680-de16cb780487" (UID: "d1178672-53f6-4fd6-a680-de16cb780487"). InnerVolumeSpecName "kube-api-access-qlhct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:08.198737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.198710    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls" pod "d1178672-53f6-4fd6-a680-de16cb780487" (UID: "d1178672-53f6-4fd6-a680-de16cb780487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:08.223236 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.223210    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:54:08.296987 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.296949    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\" (UniqueName: \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\") pod \"f72c4f05-d716-4597-abfd-ec0df8009b0e\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") "
Oct 30 21:54:08.297305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.297273    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\") pod \"f72c4f05-d716-4597-abfd-ec0df8009b0e\" (UID: \"f72c4f05-d716-4597-abfd-ec0df8009b0e\") "
Oct 30 21:54:08.297728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.297701    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume" pod "f72c4f05-d716-4597-abfd-ec0df8009b0e" (UID: "f72c4f05-d716-4597-abfd-ec0df8009b0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:08.298001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.297976    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72c4f05-d716-4597-abfd-ec0df8009b0e-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.298098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.298007    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.298098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.298017    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1178672-53f6-4fd6-a680-de16cb780487-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.298098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.298025    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhct\" (UniqueName: \"kubernetes.io/projected/d1178672-53f6-4fd6-a680-de16cb780487-kube-api-access-qlhct\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.299387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.299357    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc" pod "f72c4f05-d716-4597-abfd-ec0df8009b0e" (UID: "f72c4f05-d716-4597-abfd-ec0df8009b0e"). InnerVolumeSpecName "kube-api-access-gjscc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:08.399144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.399054    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjscc\" (UniqueName: \"kubernetes.io/projected/f72c4f05-d716-4597-abfd-ec0df8009b0e-kube-api-access-gjscc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:08.676077 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.675992    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:54:08.883368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.883333    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"
Oct 30 21:54:08.883882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.883395    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg" event={"ID":"d1178672-53f6-4fd6-a680-de16cb780487","Type":"ContainerDied","Data":"0d4b3e5e5234677d32aad5493844cd6d7f7647a18ef8c6377affb7456fde94ea"}
Oct 30 21:54:08.883882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.883451    7037 scope.go:122] "RemoveContainer" containerID="7732909fce5a79a11f72911c85befd433a74011455cc2412e7a9e9a43822fc37"
Oct 30 21:54:08.885503 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.885460    7037 generic.go:356] "Generic (PLEG): container finished" podID="f72c4f05-d716-4597-abfd-ec0df8009b0e" containerID="bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968" exitCode=0
Oct 30 21:54:08.885638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.885514    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" event={"ID":"f72c4f05-d716-4597-abfd-ec0df8009b0e","Type":"ContainerDied","Data":"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"}
Oct 30 21:54:08.885638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.885546    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7" event={"ID":"f72c4f05-d716-4597-abfd-ec0df8009b0e","Type":"ContainerDied","Data":"b97b564f39e640cab842e00a4eed51c10344f028b87ebd8f0f1f06a4465b8ca6"}
Oct 30 21:54:08.885638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.885604    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"
Oct 30 21:54:08.895615 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.895593    7037 scope.go:122] "RemoveContainer" containerID="bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"
Oct 30 21:54:08.908306 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.908282    7037 scope.go:122] "RemoveContainer" containerID="bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"
Oct 30 21:54:08.908598 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:08.908577    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968\": container with ID starting with bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968 not found: ID does not exist" containerID="bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"
Oct 30 21:54:08.908661 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.908605    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968"} err="failed to get container status \"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968\": rpc error: code = NotFound desc = could not find container \"bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968\": container with ID starting with bbb2ab8a80fd52c041be9b0c17313519c006ea7ece7ffc082b64b1ce3f009968 not found: ID does not exist"
Oct 30 21:54:08.937544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.937342    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:54:08.943173 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.943144    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8964ac-5dcd6df877-6ddsg"]
Oct 30 21:54:08.978657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.978621    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:54:08.981681 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:08.981653    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8964ac-5f69ddf9c9-ll5l7"]
Oct 30 21:54:10.709599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:10.709559    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1178672-53f6-4fd6-a680-de16cb780487" path="/var/lib/kubelet/pods/d1178672-53f6-4fd6-a680-de16cb780487/volumes"
Oct 30 21:54:10.710098 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:10.710069    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72c4f05-d716-4597-abfd-ec0df8009b0e" path="/var/lib/kubelet/pods/f72c4f05-d716-4597-abfd-ec0df8009b0e/volumes"
Oct 30 21:54:11.487657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:11.487618    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:54:15.110384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:15.110349    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:54:15.140999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:15.140961    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:54:20.111113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:20.111066    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing" containerID="cri-o://632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67" gracePeriod=25
Oct 30 21:54:20.141935 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:20.141894    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing" containerID="cri-o://ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c" gracePeriod=25
Oct 30 21:54:23.697883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:23.697814    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" podUID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" containerName="main" containerID="cri-o://7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb" gracePeriod=45
Oct 30 21:54:23.982282 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:23.982243    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:54:24.002084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.002053    7037 generic.go:356] "Generic (PLEG): container finished" podID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" containerID="7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb" exitCode=0
Oct 30 21:54:24.002262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.002117    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" event={"ID":"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0","Type":"ContainerDied","Data":"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"}
Oct 30 21:54:24.002262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.002170    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5" event={"ID":"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0","Type":"ContainerDied","Data":"74667a6f56b83c4353fd494687a0d68b91ac565215d2b89d27a01bd9e8d19943"}
Oct 30 21:54:24.002262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.002172    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"
Oct 30 21:54:24.002262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.002184    7037 scope.go:122] "RemoveContainer" containerID="7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"
Oct 30 21:54:24.012629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.012609    7037 scope.go:122] "RemoveContainer" containerID="7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"
Oct 30 21:54:24.012933 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:24.012910    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb\": container with ID starting with 7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb not found: ID does not exist" containerID="7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"
Oct 30 21:54:24.013005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.012943    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb"} err="failed to get container status \"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb\": rpc error: code = NotFound desc = could not find container \"7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb\": container with ID starting with 7c6a23e0807f25f98f279b360c4e7b21bac9780f65ead30beb0b486c9cfc00eb not found: ID does not exist"
Oct 30 21:54:24.016436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016413    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\" (UniqueName: \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016448    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016505    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016640 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016565    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016651    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016736    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\") pod \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\" (UID: \"de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0\") "
Oct 30 21:54:24.016856 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016814    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:24.016975 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.016852    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:24.017051 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.017003    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:24.017463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.017442    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.017580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.017472    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.017580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.017502    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.018894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.018861    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:24.018894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.018875    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:24.019032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.018911    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb" pod "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" (UID: "de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0"). InnerVolumeSpecName "kube-api-access-slbrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:24.118502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.118463    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbrb\" (UniqueName: \"kubernetes.io/projected/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-kube-api-access-slbrb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.118685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.118510    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.118685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.118522    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:24.349368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.349333    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:54:24.361600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.361553    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-203209/ipp-route-203209-kserve-798cf4d8df-zl7c5"]
Oct 30 21:54:24.709584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:24.709552    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" path="/var/lib/kubelet/pods/de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0/volumes"
Oct 30 21:54:26.513447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.513404    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" podUID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" containerName="main" containerID="cri-o://8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83" gracePeriod=45
Oct 30 21:54:26.800552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.800526    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:54:26.862858 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.862828    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.862876    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.862960    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.863015    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\" (UniqueName: \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.863044    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.863104    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\") pod \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\" (UID: \"e2eabc6d-140c-436f-9e3b-c4bbe151a06c\") "
Oct 30 21:54:26.863724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.863518    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:26.863886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.863803    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:26.864061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.864037    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:26.864258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.864237    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:26.865880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.865851    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:26.865984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.865926    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:26.866311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.866287    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf" pod "e2eabc6d-140c-436f-9e3b-c4bbe151a06c" (UID: "e2eabc6d-140c-436f-9e3b-c4bbe151a06c"). InnerVolumeSpecName "kube-api-access-7twbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:26.965153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.965115    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:26.965153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.965148    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:26.965153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.965158    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twbf\" (UniqueName: \"kubernetes.io/projected/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-kube-api-access-7twbf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:26.965432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.965167    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:26.965432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:26.965176    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2eabc6d-140c-436f-9e3b-c4bbe151a06c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:27.033266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.033167    7037 generic.go:356] "Generic (PLEG): container finished" podID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" containerID="8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83" exitCode=0
Oct 30 21:54:27.033266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.033219    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"
Oct 30 21:54:27.033538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.033394    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" event={"ID":"e2eabc6d-140c-436f-9e3b-c4bbe151a06c","Type":"ContainerDied","Data":"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"}
Oct 30 21:54:27.033538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.033433    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw" event={"ID":"e2eabc6d-140c-436f-9e3b-c4bbe151a06c","Type":"ContainerDied","Data":"2b5137c9e9062cea1f474434bf424b7a43e9fa4fb178d0600cfef45d4426a655"}
Oct 30 21:54:27.033538 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.033455    7037 scope.go:122] "RemoveContainer" containerID="8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"
Oct 30 21:54:27.048380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.048356    7037 scope.go:122] "RemoveContainer" containerID="8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"
Oct 30 21:54:27.048771 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:27.048743    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83\": container with ID starting with 8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83 not found: ID does not exist" containerID="8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"
Oct 30 21:54:27.048894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.048780    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83"} err="failed to get container status \"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83\": rpc error: code = NotFound desc = could not find container \"8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83\": container with ID starting with 8c486df477dd28c8bde3393e10ecd91319adcadaca8f7dba2cef9b7801ed0d83 not found: ID does not exist"
Oct 30 21:54:27.102596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.102563    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:54:27.110664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.110633    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f5bef0/sub-test-model-f5bef0-kserve-55f7cf55b5-jsthw"]
Oct 30 21:54:27.369909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.369869    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:27.370094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:27.369911    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:28.710633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:28.710601    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" path="/var/lib/kubelet/pods/e2eabc6d-140c-436f-9e3b-c4bbe151a06c/volumes"
Oct 30 21:54:33.674841 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:33.674794    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:54:37.378511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:37.375360    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:37.378511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:37.376050    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:40.439885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.439857    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:54:40.468672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.468648    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:54:40.590647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.590609    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\") pod \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") "
Oct 30 21:54:40.590830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.590680    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\" (UniqueName: \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\") pod \"f27835da-2169-4319-aad7-bf998a74c2e0\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") "
Oct 30 21:54:40.590830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.590719    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\" (UniqueName: \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\") pod \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\" (UID: \"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f\") "
Oct 30 21:54:40.590830 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.590748    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\") pod \"f27835da-2169-4319-aad7-bf998a74c2e0\" (UID: \"f27835da-2169-4319-aad7-bf998a74c2e0\") "
Oct 30 21:54:40.591109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.591085    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume" pod "ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" (UID: "ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:40.591183 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.591165    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume" pod "f27835da-2169-4319-aad7-bf998a74c2e0" (UID: "f27835da-2169-4319-aad7-bf998a74c2e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:40.591471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.591456    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:40.591556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.591474    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27835da-2169-4319-aad7-bf998a74c2e0-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:40.592923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.592902    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d" pod "ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" (UID: "ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f"). InnerVolumeSpecName "kube-api-access-ff26d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.592972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.592932    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42" pod "f27835da-2169-4319-aad7-bf998a74c2e0" (UID: "f27835da-2169-4319-aad7-bf998a74c2e0"). InnerVolumeSpecName "kube-api-access-qdf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.693262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.693186    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdf42\" (UniqueName: \"kubernetes.io/projected/f27835da-2169-4319-aad7-bf998a74c2e0-kube-api-access-qdf42\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:40.693262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:40.693215    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff26d\" (UniqueName: \"kubernetes.io/projected/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f-kube-api-access-ff26d\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:41.142878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.142777    7037 generic.go:356] "Generic (PLEG): container finished" podID="f27835da-2169-4319-aad7-bf998a74c2e0" containerID="632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67" exitCode=0
Oct 30 21:54:41.143149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.142901    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"
Oct 30 21:54:41.143149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.143043    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" event={"ID":"f27835da-2169-4319-aad7-bf998a74c2e0","Type":"ContainerDied","Data":"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"}
Oct 30 21:54:41.143149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.143090    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq" event={"ID":"f27835da-2169-4319-aad7-bf998a74c2e0","Type":"ContainerDied","Data":"cbfc574a9ca094ca4bac9db52f6b88aa79de386f7792cae2623222a8c9a2adc8"}
Oct 30 21:54:41.143149 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.143115    7037 scope.go:122] "RemoveContainer" containerID="632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"
Oct 30 21:54:41.145444 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.145414    7037 generic.go:356] "Generic (PLEG): container finished" podID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerID="ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c" exitCode=0
Oct 30 21:54:41.145771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.145469    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" event={"ID":"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f","Type":"ContainerDied","Data":"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"}
Oct 30 21:54:41.145771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.145559    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f" event={"ID":"ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f","Type":"ContainerDied","Data":"5f58702d67e75a4be9020ec73409e25d5149f5cf0ca5829d282f302bfdb501e2"}
Oct 30 21:54:41.145771 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.145652    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"
Oct 30 21:54:41.157964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.157942    7037 scope.go:122] "RemoveContainer" containerID="632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"
Oct 30 21:54:41.158342 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:41.158315    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67\": container with ID starting with 632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67 not found: ID does not exist" containerID="632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"
Oct 30 21:54:41.158466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.158351    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67"} err="failed to get container status \"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67\": rpc error: code = NotFound desc = could not find container \"632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67\": container with ID starting with 632bfc47dda0f784452320e5b995101459c263deb6c002240081e6ee360fba67 not found: ID does not exist"
Oct 30 21:54:41.158466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.158374    7037 scope.go:122] "RemoveContainer" containerID="ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"
Oct 30 21:54:41.169873 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.169844    7037 scope.go:122] "RemoveContainer" containerID="ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"
Oct 30 21:54:41.170222 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:41.170202    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c\": container with ID starting with ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c not found: ID does not exist" containerID="ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"
Oct 30 21:54:41.170329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.170227    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c"} err="failed to get container status \"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c\": rpc error: code = NotFound desc = could not find container \"ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c\": container with ID starting with ecc6adc5430509ebdedf373845c436e264acea1e6dcc6de88095f440a1f2f61c not found: ID does not exist"
Oct 30 21:54:41.195648 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.195606    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:54:41.207685 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.207653    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ad73a4-5b685558f8-npfkq"]
Oct 30 21:54:41.242628 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.242587    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:54:41.245414 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:41.245390    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ad73a4-77f578f55-vhv5f"]
Oct 30 21:54:42.708657 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:42.708623    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" path="/var/lib/kubelet/pods/ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f/volumes"
Oct 30 21:54:42.709140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:42.709100    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" path="/var/lib/kubelet/pods/f27835da-2169-4319-aad7-bf998a74c2e0/volumes"
Oct 30 21:54:46.553757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:46.553714    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:54:46.586087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:46.586053    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:54:47.303060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.303025    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:54:47.375399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.375366    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:54:47.375927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.375876    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" podUID="ab275431-00d5-4a8f-8de4-f780fde9d792" containerName="maas-api" containerID="cri-o://aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d" gracePeriod=30
Oct 30 21:54:47.683881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.683846    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:54:47.781792 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.781754    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\") pod \"ab275431-00d5-4a8f-8de4-f780fde9d792\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") "
Oct 30 21:54:47.782023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.781824    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\" (UniqueName: \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\") pod \"ab275431-00d5-4a8f-8de4-f780fde9d792\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") "
Oct 30 21:54:47.782023 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.781974    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\") pod \"ab275431-00d5-4a8f-8de4-f780fde9d792\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") "
Oct 30 21:54:47.782208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.782185    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") pod \"ab275431-00d5-4a8f-8de4-f780fde9d792\" (UID: \"ab275431-00d5-4a8f-8de4-f780fde9d792\") "
Oct 30 21:54:47.782396 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.782367    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca" pod "ab275431-00d5-4a8f-8de4-f780fde9d792" (UID: "ab275431-00d5-4a8f-8de4-f780fde9d792"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:47.783621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.783593    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab275431-00d5-4a8f-8de4-f780fde9d792-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:47.784522 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.784464    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl" pod "ab275431-00d5-4a8f-8de4-f780fde9d792" (UID: "ab275431-00d5-4a8f-8de4-f780fde9d792"). InnerVolumeSpecName "kube-api-access-7lsfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.784646 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.784627    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs" pod "ab275431-00d5-4a8f-8de4-f780fde9d792" (UID: "ab275431-00d5-4a8f-8de4-f780fde9d792"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:47.784691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.784669    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls" pod "ab275431-00d5-4a8f-8de4-f780fde9d792" (UID: "ab275431-00d5-4a8f-8de4-f780fde9d792"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:47.884484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.884444    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:47.884484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.884481    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lsfl\" (UniqueName: \"kubernetes.io/projected/ab275431-00d5-4a8f-8de4-f780fde9d792-kube-api-access-7lsfl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:47.884484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:47.884512    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab275431-00d5-4a8f-8de4-f780fde9d792-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:48.203843 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.203751    7037 generic.go:356] "Generic (PLEG): container finished" podID="ab275431-00d5-4a8f-8de4-f780fde9d792" containerID="aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d" exitCode=0
Oct 30 21:54:48.204032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.203850    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"
Oct 30 21:54:48.204032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.203882    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" event={"ID":"ab275431-00d5-4a8f-8de4-f780fde9d792","Type":"ContainerDied","Data":"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"}
Oct 30 21:54:48.204032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.203953    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m" event={"ID":"ab275431-00d5-4a8f-8de4-f780fde9d792","Type":"ContainerDied","Data":"d40935ee4cd302175680669bf6d2c3ce1a760e80d178af1c98ef87a1267d2b72"}
Oct 30 21:54:48.204032 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.204001    7037 scope.go:122] "RemoveContainer" containerID="aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"
Oct 30 21:54:48.219034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.219013    7037 scope.go:122] "RemoveContainer" containerID="aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"
Oct 30 21:54:48.219392 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:48.219365    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d\": container with ID starting with aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d not found: ID does not exist" containerID="aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"
Oct 30 21:54:48.219476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.219405    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d"} err="failed to get container status \"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d\": rpc error: code = NotFound desc = could not find container \"aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d\": container with ID starting with aa7e233e532eb9364f772db0e65fcc621a14da59480863ed63e4a664a31e490d not found: ID does not exist"
Oct 30 21:54:48.263358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.263305    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:54:48.267004 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.266981    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-70d3a9-6947d84998-vhn9m"]
Oct 30 21:54:48.534844 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.534756    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:54:48.713760 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:48.713226    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab275431-00d5-4a8f-8de4-f780fde9d792" path="/var/lib/kubelet/pods/ab275431-00d5-4a8f-8de4-f780fde9d792/volumes"
Oct 30 21:54:51.538019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.537985    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 21:54:51.538688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538674    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ab275431-00d5-4a8f-8de4-f780fde9d792" containerName="maas-api"
Oct 30 21:54:51.538748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538689    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab275431-00d5-4a8f-8de4-f780fde9d792" containerName="maas-api"
Oct 30 21:54:51.538748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538726    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7c29cfd3-fe68-4727-a98b-5408248231c2" containerName="maas-api"
Oct 30 21:54:51.538748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538731    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c29cfd3-fe68-4727-a98b-5408248231c2" containerName="maas-api"
Oct 30 21:54:51.538748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538740    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d1178672-53f6-4fd6-a680-de16cb780487" containerName="maas-api"
Oct 30 21:54:51.538748 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538746    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1178672-53f6-4fd6-a680-de16cb780487" containerName="maas-api"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538760    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538765    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538772    7037 cpu_manager.go:414] "containerMap: removing container" podUID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" containerName="maas-api"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538776    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" containerName="maas-api"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538789    7037 cpu_manager.go:414] "containerMap: removing container" podUID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" containerName="main"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538794    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" containerName="main"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538804    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" containerName="main"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538808    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" containerName="main"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538815    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538819    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538827    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5080b2dd-7157-453b-9323-90cbbb50e6ac" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538832    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5080b2dd-7157-453b-9323-90cbbb50e6ac" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538839    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f72c4f05-d716-4597-abfd-ec0df8009b0e" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538844    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72c4f05-d716-4597-abfd-ec0df8009b0e" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538851    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" containerName="payload-pre-processing"
Oct 30 21:54:51.538922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538855    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" containerName="payload-pre-processing"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538945    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c568bd-dbd0-4164-9e2c-1b36b1cd90d0" containerName="maas-api"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538953    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6dc9d8-15ef-45fa-8a70-520dfa0fe8e0" containerName="main"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538959    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2eabc6d-140c-436f-9e3b-c4bbe151a06c" containerName="main"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538969    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76e62d-ce9f-4f42-8a7f-c7e8ddd1314f" containerName="payload-pre-processing"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538976    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5080b2dd-7157-453b-9323-90cbbb50e6ac" containerName="payload-pre-processing"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538983    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72c4f05-d716-4597-abfd-ec0df8009b0e" containerName="payload-pre-processing"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538990    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1168bc-7bd1-42bc-8c74-b4e700b46377" containerName="payload-pre-processing"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.538998    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab275431-00d5-4a8f-8de4-f780fde9d792" containerName="maas-api"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.539008    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c29cfd3-fe68-4727-a98b-5408248231c2" containerName="maas-api"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.539016    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1178672-53f6-4fd6-a680-de16cb780487" containerName="maas-api"
Oct 30 21:54:51.539420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.539023    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27835da-2169-4319-aad7-bf998a74c2e0" containerName="payload-processing"
Oct 30 21:54:51.543899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.543878    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.554574 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.554547    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing" containerID="cri-o://9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb" gracePeriod=25
Oct 30 21:54:51.569724 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.569699    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:54:51.573753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.573732    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:54:51.573854 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.573759    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:54:51.574031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.574005    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wmcm5\""
Oct 30 21:54:51.594533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.586336    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" podUID="9e24018a-efff-4258-aebd-b3722015e5d3" containerName="payload-pre-processing" containerID="cri-o://c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28" gracePeriod=25
Oct 30 21:54:51.594533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.591015    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 21:54:51.636343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636309    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636423    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636463    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636510    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636699 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636645    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqxh\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636759 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636697    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.636866 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.636845    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738058 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738024    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqxh\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738085    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738141    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738198    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738274 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738249    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738292    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.738536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.738515    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.739151 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.739091    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.741899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.741869    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.742037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.741969    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.742208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.742189    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.743693 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.743674    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.760726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.760693    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.761356 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.761332    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqxh\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzl4j\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.857709 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.857679    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:51.866921 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.866897    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:54:51.941174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.941134    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\" (UniqueName: \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\") pod \"9e24018a-efff-4258-aebd-b3722015e5d3\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") "
Oct 30 21:54:51.941320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.941245    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\") pod \"9e24018a-efff-4258-aebd-b3722015e5d3\" (UID: \"9e24018a-efff-4258-aebd-b3722015e5d3\") "
Oct 30 21:54:51.941890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.941864    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume" pod "9e24018a-efff-4258-aebd-b3722015e5d3" (UID: "9e24018a-efff-4258-aebd-b3722015e5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:51.943229 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:51.943206    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv" pod "9e24018a-efff-4258-aebd-b3722015e5d3" (UID: "9e24018a-efff-4258-aebd-b3722015e5d3"). InnerVolumeSpecName "kube-api-access-rtwnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:52.043203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.043168    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwnv\" (UniqueName: \"kubernetes.io/projected/9e24018a-efff-4258-aebd-b3722015e5d3-kube-api-access-rtwnv\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:52.043203 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.043200    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e24018a-efff-4258-aebd-b3722015e5d3-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:54:52.184362 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:54:52.184324    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb443c850_ad96_445e_9e45_538af32948b0.slice/crio-8ea975b059a8293a83460a6d7da4c0f809bf8b0c7afb7c417a60d162e6d31377 WatchSource:0}: Error finding container 8ea975b059a8293a83460a6d7da4c0f809bf8b0c7afb7c417a60d162e6d31377: Status 404 returned error can't find the container with id 8ea975b059a8293a83460a6d7da4c0f809bf8b0c7afb7c417a60d162e6d31377
Oct 30 21:54:52.185145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.185118    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 21:54:52.187242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.187205    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:54:52.187316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.187271    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 21:54:52.237181 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.237150    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" event={"ID":"b443c850-ad96-445e-9e45-538af32948b0","Type":"ContainerStarted","Data":"8ea975b059a8293a83460a6d7da4c0f809bf8b0c7afb7c417a60d162e6d31377"}
Oct 30 21:54:52.238674 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.238647    7037 generic.go:356] "Generic (PLEG): container finished" podID="9e24018a-efff-4258-aebd-b3722015e5d3" containerID="c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28" exitCode=0
Oct 30 21:54:52.238799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.238681    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" event={"ID":"9e24018a-efff-4258-aebd-b3722015e5d3","Type":"ContainerDied","Data":"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"}
Oct 30 21:54:52.238799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.238694    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"
Oct 30 21:54:52.238799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.238713    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj" event={"ID":"9e24018a-efff-4258-aebd-b3722015e5d3","Type":"ContainerDied","Data":"44adeb047d450d47fc82d41d5181c031c213cdaa13f9bda15c1adfcf1a889b33"}
Oct 30 21:54:52.238799 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.238733    7037 scope.go:122] "RemoveContainer" containerID="c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"
Oct 30 21:54:52.250244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.250221    7037 scope.go:122] "RemoveContainer" containerID="c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"
Oct 30 21:54:52.250554 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:54:52.250533    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28\": container with ID starting with c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28 not found: ID does not exist" containerID="c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"
Oct 30 21:54:52.250620 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.250567    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28"} err="failed to get container status \"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28\": rpc error: code = NotFound desc = could not find container \"c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28\": container with ID starting with c963ca7a79e515da0130c4722085500e540196adeef827fbf65ee64c00521e28 not found: ID does not exist"
Oct 30 21:54:52.299524 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.296558    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:54:52.300723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.300692    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-70d3a9-768b7dbd8c-znkvj"]
Oct 30 21:54:52.304321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.304288    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing" containerID="cri-o://bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4" gracePeriod=25
Oct 30 21:54:52.709621 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:52.709579    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e24018a-efff-4258-aebd-b3722015e5d3" path="/var/lib/kubelet/pods/9e24018a-efff-4258-aebd-b3722015e5d3/volumes"
Oct 30 21:54:53.252391 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.252345    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" event={"ID":"b443c850-ad96-445e-9e45-538af32948b0","Type":"ContainerStarted","Data":"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"}
Oct 30 21:54:53.252868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.252843    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:53.255111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.255081    7037 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzl4j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:53.255240 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.255130    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" podUID="b443c850-ad96-445e-9e45-538af32948b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:53.362639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.362580    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" podStartSLOduration=2.362561583 podStartE2EDuration="2.362561583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:53.345928638 +0000 UTC m=+3236.757468062" watchObservedRunningTime="2026-10-30 21:54:53.362561583 +0000 UTC m=+3236.774101031"
Oct 30 21:54:53.535974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:53.535872    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing" containerID="cri-o://ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0" gracePeriod=25
Oct 30 21:54:54.261979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:54.261949    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 21:54:56.742468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:56.742430    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:56.899790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:54:56.899721    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:02.990916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:02.990867    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:03.674132 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:03.674092    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:55:06.743099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:06.743016    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:06.899938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:06.899893    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:11.836735 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.836703    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:55:11.919442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.919404    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\" (UniqueName: \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\") pod \"696591a5-faa5-4462-9994-292c599c6030\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") "
Oct 30 21:55:11.919671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.919608    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\") pod \"696591a5-faa5-4462-9994-292c599c6030\" (UID: \"696591a5-faa5-4462-9994-292c599c6030\") "
Oct 30 21:55:11.920065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.920028    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume" pod "696591a5-faa5-4462-9994-292c599c6030" (UID: "696591a5-faa5-4462-9994-292c599c6030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:11.920209 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.920189    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/696591a5-faa5-4462-9994-292c599c6030-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:11.921838 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:11.921805    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4" pod "696591a5-faa5-4462-9994-292c599c6030" (UID: "696591a5-faa5-4462-9994-292c599c6030"). InnerVolumeSpecName "kube-api-access-86bf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:12.021994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.021959    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86bf4\" (UniqueName: \"kubernetes.io/projected/696591a5-faa5-4462-9994-292c599c6030-kube-api-access-86bf4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:12.418934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.418893    7037 generic.go:356] "Generic (PLEG): container finished" podID="696591a5-faa5-4462-9994-292c599c6030" containerID="9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb" exitCode=0
Oct 30 21:55:12.419108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.419026    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" event={"ID":"696591a5-faa5-4462-9994-292c599c6030","Type":"ContainerDied","Data":"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"}
Oct 30 21:55:12.419108 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.419080    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5" event={"ID":"696591a5-faa5-4462-9994-292c599c6030","Type":"ContainerDied","Data":"e760329335b735261b4288dfc96ff8e3022f9e919d5e1e998b95aef2a15b06e2"}
Oct 30 21:55:12.419250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.419111    7037 scope.go:122] "RemoveContainer" containerID="9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"
Oct 30 21:55:12.419315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.419298    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"
Oct 30 21:55:12.434184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.434150    7037 scope.go:122] "RemoveContainer" containerID="9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"
Oct 30 21:55:12.434842 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:12.434813    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb\": container with ID starting with 9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb not found: ID does not exist" containerID="9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"
Oct 30 21:55:12.434950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.434853    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb"} err="failed to get container status \"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb\": rpc error: code = NotFound desc = could not find container \"9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb\": container with ID starting with 9f91891a108d6a3b6b3afcc70a0523eee278f0b6a4ce538546dfaf68b6551eeb not found: ID does not exist"
Oct 30 21:55:12.482717 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.482678    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:55:12.500440 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.500403    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-70d3a9-676bf9fd96-wt7d5"]
Oct 30 21:55:12.567410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.567389    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:55:12.629664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.629634    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\" (UniqueName: \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\") pod \"c7d67858-28fe-4705-9b95-567c6d9acc57\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") "
Oct 30 21:55:12.629845 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.629732    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\") pod \"c7d67858-28fe-4705-9b95-567c6d9acc57\" (UID: \"c7d67858-28fe-4705-9b95-567c6d9acc57\") "
Oct 30 21:55:12.630200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.630169    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume" pod "c7d67858-28fe-4705-9b95-567c6d9acc57" (UID: "c7d67858-28fe-4705-9b95-567c6d9acc57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:12.632138 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.632109    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb" pod "c7d67858-28fe-4705-9b95-567c6d9acc57" (UID: "c7d67858-28fe-4705-9b95-567c6d9acc57"). InnerVolumeSpecName "kube-api-access-4zczb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:12.708065 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.708030    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696591a5-faa5-4462-9994-292c599c6030" path="/var/lib/kubelet/pods/696591a5-faa5-4462-9994-292c599c6030/volumes"
Oct 30 21:55:12.731031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.730998    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zczb\" (UniqueName: \"kubernetes.io/projected/c7d67858-28fe-4705-9b95-567c6d9acc57-kube-api-access-4zczb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:12.731031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.731026    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7d67858-28fe-4705-9b95-567c6d9acc57-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:12.990814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:12.990719    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:13.249803 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.249653    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:55:13.251277 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.251166    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" podUID="967aa72f-f0ad-4f40-8559-37a251c093c3" containerName="maas-api" containerID="cri-o://a45498134c55275d7e1557a19fbd7c46803785e4431654fcee37a4ffad71911e" gracePeriod=30
Oct 30 21:55:13.428683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.428649    7037 generic.go:356] "Generic (PLEG): container finished" podID="967aa72f-f0ad-4f40-8559-37a251c093c3" containerID="a45498134c55275d7e1557a19fbd7c46803785e4431654fcee37a4ffad71911e" exitCode=0
Oct 30 21:55:13.428857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.428723    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" event={"ID":"967aa72f-f0ad-4f40-8559-37a251c093c3","Type":"ContainerDied","Data":"a45498134c55275d7e1557a19fbd7c46803785e4431654fcee37a4ffad71911e"}
Oct 30 21:55:13.430290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.430267    7037 generic.go:356] "Generic (PLEG): container finished" podID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerID="bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4" exitCode=0
Oct 30 21:55:13.430400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.430316    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" event={"ID":"c7d67858-28fe-4705-9b95-567c6d9acc57","Type":"ContainerDied","Data":"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"}
Oct 30 21:55:13.430400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.430322    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"
Oct 30 21:55:13.430400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.430358    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx" event={"ID":"c7d67858-28fe-4705-9b95-567c6d9acc57","Type":"ContainerDied","Data":"b1c74c5127771dc61664c21aaed41df59870b8a264f89534e4a336a64119b33f"}
Oct 30 21:55:13.430400 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.430380    7037 scope.go:122] "RemoveContainer" containerID="bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"
Oct 30 21:55:13.441663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.441632    7037 scope.go:122] "RemoveContainer" containerID="bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"
Oct 30 21:55:13.442016 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:13.441989    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4\": container with ID starting with bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4 not found: ID does not exist" containerID="bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"
Oct 30 21:55:13.442093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.442027    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4"} err="failed to get container status \"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4\": rpc error: code = NotFound desc = could not find container \"bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4\": container with ID starting with bb83d6a48b21955b9f8a90b5e95998bc67af0e814973ce84612b60c54ec233b4 not found: ID does not exist"
Oct 30 21:55:13.525658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.516572    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:55:13.525658 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.520685    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f5bef0-6898c5fb7-kf5jx"]
Oct 30 21:55:13.685787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.685758    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:55:13.745141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.745105    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\") pod \"967aa72f-f0ad-4f40-8559-37a251c093c3\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") "
Oct 30 21:55:13.745346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.745206    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\") pod \"967aa72f-f0ad-4f40-8559-37a251c093c3\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") "
Oct 30 21:55:13.745346 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.745336    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\" (UniqueName: \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\") pod \"967aa72f-f0ad-4f40-8559-37a251c093c3\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") "
Oct 30 21:55:13.745533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.745512    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\") pod \"967aa72f-f0ad-4f40-8559-37a251c093c3\" (UID: \"967aa72f-f0ad-4f40-8559-37a251c093c3\") "
Oct 30 21:55:13.745946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.745916    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca" pod "967aa72f-f0ad-4f40-8559-37a251c093c3" (UID: "967aa72f-f0ad-4f40-8559-37a251c093c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:13.746442 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.746235    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967aa72f-f0ad-4f40-8559-37a251c093c3-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:13.748532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.747952    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs" pod "967aa72f-f0ad-4f40-8559-37a251c093c3" (UID: "967aa72f-f0ad-4f40-8559-37a251c093c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:13.750627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.748913    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls" pod "967aa72f-f0ad-4f40-8559-37a251c093c3" (UID: "967aa72f-f0ad-4f40-8559-37a251c093c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:13.750627 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.750588    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr" pod "967aa72f-f0ad-4f40-8559-37a251c093c3" (UID: "967aa72f-f0ad-4f40-8559-37a251c093c3"). InnerVolumeSpecName "kube-api-access-gsqtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:13.847339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.847306    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:13.847339 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.847334    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967aa72f-f0ad-4f40-8559-37a251c093c3-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:13.847548 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.847345    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqtr\" (UniqueName: \"kubernetes.io/projected/967aa72f-f0ad-4f40-8559-37a251c093c3-kube-api-access-gsqtr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:13.849337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.849309    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:55:13.947961 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.947922    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\") pod \"68ddf27d-ae2e-4c81-9153-060f55931652\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") "
Oct 30 21:55:13.948180 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.947997    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\" (UniqueName: \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\") pod \"68ddf27d-ae2e-4c81-9153-060f55931652\" (UID: \"68ddf27d-ae2e-4c81-9153-060f55931652\") "
Oct 30 21:55:13.948405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.948380    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume" pod "68ddf27d-ae2e-4c81-9153-060f55931652" (UID: "68ddf27d-ae2e-4c81-9153-060f55931652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:13.950127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:13.950105    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv" pod "68ddf27d-ae2e-4c81-9153-060f55931652" (UID: "68ddf27d-ae2e-4c81-9153-060f55931652"). InnerVolumeSpecName "kube-api-access-52rwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:14.049989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.049950    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ddf27d-ae2e-4c81-9153-060f55931652-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:14.049989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.049981    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rwv\" (UniqueName: \"kubernetes.io/projected/68ddf27d-ae2e-4c81-9153-060f55931652-kube-api-access-52rwv\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:14.437976 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.437943    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"
Oct 30 21:55:14.438202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.437984    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb" event={"ID":"967aa72f-f0ad-4f40-8559-37a251c093c3","Type":"ContainerDied","Data":"c17b004333123f08885672752d84bb7bd225292d4f36f8b8f4fa44098f52ceef"}
Oct 30 21:55:14.438202 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.438079    7037 scope.go:122] "RemoveContainer" containerID="a45498134c55275d7e1557a19fbd7c46803785e4431654fcee37a4ffad71911e"
Oct 30 21:55:14.440757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.440729    7037 generic.go:356] "Generic (PLEG): container finished" podID="68ddf27d-ae2e-4c81-9153-060f55931652" containerID="ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0" exitCode=0
Oct 30 21:55:14.440883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.440782    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"
Oct 30 21:55:14.440883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.440777    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" event={"ID":"68ddf27d-ae2e-4c81-9153-060f55931652","Type":"ContainerDied","Data":"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"}
Oct 30 21:55:14.440883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.440875    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb" event={"ID":"68ddf27d-ae2e-4c81-9153-060f55931652","Type":"ContainerDied","Data":"ae32fea308397b1866f90a22b8c0fdd6e435c58f5583193fd11c1fe6558dcbea"}
Oct 30 21:55:14.449964 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.449908    7037 scope.go:122] "RemoveContainer" containerID="ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"
Oct 30 21:55:14.460069 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.460049    7037 scope.go:122] "RemoveContainer" containerID="ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"
Oct 30 21:55:14.460336 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:14.460318    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0\": container with ID starting with ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0 not found: ID does not exist" containerID="ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"
Oct 30 21:55:14.460380 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.460346    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0"} err="failed to get container status \"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0\": rpc error: code = NotFound desc = could not find container \"ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0\": container with ID starting with ffdbe4cab6377c7154f9af757aaec72d44dbf6ce53ea8ff5a40ad61b6b101da0 not found: ID does not exist"
Oct 30 21:55:14.507056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.507021    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:55:14.512582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.512551    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f5bef0-67f47984f9-f7phb"]
Oct 30 21:55:14.549336 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.549294    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:55:14.559501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.559455    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-66cc54-96bc8fd5d-fnsvb"]
Oct 30 21:55:14.708879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.708805    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" path="/var/lib/kubelet/pods/68ddf27d-ae2e-4c81-9153-060f55931652/volumes"
Oct 30 21:55:14.709314 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.709300    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967aa72f-f0ad-4f40-8559-37a251c093c3" path="/var/lib/kubelet/pods/967aa72f-f0ad-4f40-8559-37a251c093c3/volumes"
Oct 30 21:55:14.709726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:14.709712    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" path="/var/lib/kubelet/pods/c7d67858-28fe-4705-9b95-567c6d9acc57/volumes"
Oct 30 21:55:25.152590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:25.152556    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:55:30.153063 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:30.153015    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing" containerID="cri-o://4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a" gracePeriod=25
Oct 30 21:55:33.009566 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:33.009521    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:55:33.675562 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:33.675519    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:55:34.379225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.379191    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:55:34.379865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.379414    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" podUID="856ee13a-a04c-468f-a277-163bf956e0de" containerName="payload-pre-processing" containerID="cri-o://d296ee51c3df44e82e9a58e93f99fd4ed8ee9c4ca1396b664de78a825a40a388" gracePeriod=30
Oct 30 21:55:34.598092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.590755    7037 generic.go:356] "Generic (PLEG): container finished" podID="856ee13a-a04c-468f-a277-163bf956e0de" containerID="d296ee51c3df44e82e9a58e93f99fd4ed8ee9c4ca1396b664de78a825a40a388" exitCode=0
Oct 30 21:55:34.598092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.590839    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" event={"ID":"856ee13a-a04c-468f-a277-163bf956e0de","Type":"ContainerDied","Data":"d296ee51c3df44e82e9a58e93f99fd4ed8ee9c4ca1396b664de78a825a40a388"}
Oct 30 21:55:34.611259 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.611222    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:34.666465 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.666434    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:55:34.761851 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.761803    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\") pod \"856ee13a-a04c-468f-a277-163bf956e0de\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") "
Oct 30 21:55:34.762092 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.762002    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\" (UniqueName: \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\") pod \"856ee13a-a04c-468f-a277-163bf956e0de\" (UID: \"856ee13a-a04c-468f-a277-163bf956e0de\") "
Oct 30 21:55:34.763782 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.763721    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume" pod "856ee13a-a04c-468f-a277-163bf956e0de" (UID: "856ee13a-a04c-468f-a277-163bf956e0de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:34.767594 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.765325    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn" pod "856ee13a-a04c-468f-a277-163bf956e0de" (UID: "856ee13a-a04c-468f-a277-163bf956e0de"). InnerVolumeSpecName "kube-api-access-7gmtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:34.863464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.863426    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmtn\" (UniqueName: \"kubernetes.io/projected/856ee13a-a04c-468f-a277-163bf956e0de-kube-api-access-7gmtn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:34.863464 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:34.863459    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856ee13a-a04c-468f-a277-163bf956e0de-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:35.600166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:35.600129    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"
Oct 30 21:55:35.601564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:35.600028    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx" event={"ID":"856ee13a-a04c-468f-a277-163bf956e0de","Type":"ContainerDied","Data":"4b6beaeb5fe9f5e9a64ddc19aa818bc1c2efa82e445f5353236974f8321266e2"}
Oct 30 21:55:35.601705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:35.601635    7037 scope.go:122] "RemoveContainer" containerID="d296ee51c3df44e82e9a58e93f99fd4ed8ee9c4ca1396b664de78a825a40a388"
Oct 30 21:55:35.659066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:35.659023    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:55:35.670644 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:35.670607    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e60e16-785ff85447-s94wx"]
Oct 30 21:55:36.708945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:36.708913    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856ee13a-a04c-468f-a277-163bf956e0de" path="/var/lib/kubelet/pods/856ee13a-a04c-468f-a277-163bf956e0de/volumes"
Oct 30 21:55:38.010028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:38.009984    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing" containerID="cri-o://ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17" gracePeriod=25
Oct 30 21:55:44.611322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:44.611273    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:47.350560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.350518    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:47.567224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.567163    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:55:47.567530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.567463    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" podUID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" containerName="maas-api" containerID="cri-o://66c037b42bf728beb34fcea361787eaab7b23567735e3b24472a76b5b45529b1" gracePeriod=30
Oct 30 21:55:47.700280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.700248    7037 generic.go:356] "Generic (PLEG): container finished" podID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" containerID="66c037b42bf728beb34fcea361787eaab7b23567735e3b24472a76b5b45529b1" exitCode=0
Oct 30 21:55:47.700469 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.700310    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" event={"ID":"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc","Type":"ContainerDied","Data":"66c037b42bf728beb34fcea361787eaab7b23567735e3b24472a76b5b45529b1"}
Oct 30 21:55:47.894947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:47.894913    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:55:48.045814 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.045729    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\") pod \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") "
Oct 30 21:55:48.045978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.045939    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\" (UniqueName: \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\") pod \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") "
Oct 30 21:55:48.046044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.045982    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\") pod \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") "
Oct 30 21:55:48.046044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.046002    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\") pod \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\" (UID: \"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc\") "
Oct 30 21:55:48.046166 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.046083    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca" pod "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" (UID: "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:48.046537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.046513    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:48.048168 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.048139    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls" pod "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" (UID: "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:48.048267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.048213    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs" pod "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" (UID: "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:48.048267 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.048228    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd" pod "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" (UID: "f55e7476-3fb6-41e5-ad6e-d7b13bc414cc"). InnerVolumeSpecName "kube-api-access-kxwwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:48.147427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.147387    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwwd\" (UniqueName: \"kubernetes.io/projected/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-kube-api-access-kxwwd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:48.147427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.147420    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:48.147427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.147432    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:48.710132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.710098    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"
Oct 30 21:55:48.710695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.710132    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f" event={"ID":"f55e7476-3fb6-41e5-ad6e-d7b13bc414cc","Type":"ContainerDied","Data":"5c14779624195bcc471ee7b506a0c130bf55c7cbea334854f11e048886e6565a"}
Oct 30 21:55:48.710695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.710207    7037 scope.go:122] "RemoveContainer" containerID="66c037b42bf728beb34fcea361787eaab7b23567735e3b24472a76b5b45529b1"
Oct 30 21:55:48.776593 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.776546    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:55:48.788083 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:48.788050    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9cd83f-6898787cbd-h9f6f"]
Oct 30 21:55:50.436520 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.436461    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:55:50.479591 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.479555    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\" (UniqueName: \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\") pod \"318c1981-a3b1-4528-9229-c46921459fb4\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") "
Oct 30 21:55:50.479794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.479742    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\") pod \"318c1981-a3b1-4528-9229-c46921459fb4\" (UID: \"318c1981-a3b1-4528-9229-c46921459fb4\") "
Oct 30 21:55:50.480536 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.480461    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume" pod "318c1981-a3b1-4528-9229-c46921459fb4" (UID: "318c1981-a3b1-4528-9229-c46921459fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:50.482564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.482534    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj" pod "318c1981-a3b1-4528-9229-c46921459fb4" (UID: "318c1981-a3b1-4528-9229-c46921459fb4"). InnerVolumeSpecName "kube-api-access-klbkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:50.581925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.581890    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbkj\" (UniqueName: \"kubernetes.io/projected/318c1981-a3b1-4528-9229-c46921459fb4-kube-api-access-klbkj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:50.581925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.581922    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318c1981-a3b1-4528-9229-c46921459fb4-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:50.708301 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.708222    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" path="/var/lib/kubelet/pods/f55e7476-3fb6-41e5-ad6e-d7b13bc414cc/volumes"
Oct 30 21:55:50.725900 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.725866    7037 generic.go:356] "Generic (PLEG): container finished" podID="318c1981-a3b1-4528-9229-c46921459fb4" containerID="4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a" exitCode=0
Oct 30 21:55:50.726059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.725925    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"
Oct 30 21:55:50.726059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.725945    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" event={"ID":"318c1981-a3b1-4528-9229-c46921459fb4","Type":"ContainerDied","Data":"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"}
Oct 30 21:55:50.726059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.726006    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq" event={"ID":"318c1981-a3b1-4528-9229-c46921459fb4","Type":"ContainerDied","Data":"a86138890278f3d33995ba304810fdca1541aca3a0acb36b5ed5bf33c8ea4cb8"}
Oct 30 21:55:50.726059 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.726036    7037 scope.go:122] "RemoveContainer" containerID="4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"
Oct 30 21:55:50.736660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.736636    7037 scope.go:122] "RemoveContainer" containerID="4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"
Oct 30 21:55:50.736939 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:50.736914    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a\": container with ID starting with 4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a not found: ID does not exist" containerID="4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"
Oct 30 21:55:50.737049 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.736945    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a"} err="failed to get container status \"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a\": rpc error: code = NotFound desc = could not find container \"4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a\": container with ID starting with 4c7452e8b17506d0e9fbbf92d583b2a99aea02056e8d3cf615bc5a01fcb1c46a not found: ID does not exist"
Oct 30 21:55:50.786992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.786958    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:55:50.792086 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:50.792058    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9cd83f-86c464f86c-l45bq"]
Oct 30 21:55:52.708337 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:52.708288    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318c1981-a3b1-4528-9229-c46921459fb4" path="/var/lib/kubelet/pods/318c1981-a3b1-4528-9229-c46921459fb4/volumes"
Oct 30 21:55:53.151908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.151875    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:55:53.152127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.152102    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" podUID="02d09b60-e573-40c6-9d22-bdd339728f14" containerName="maas-api" containerID="cri-o://9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6" gracePeriod=30
Oct 30 21:55:53.446020 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.445947    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:55:53.523207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.523174    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\") pod \"02d09b60-e573-40c6-9d22-bdd339728f14\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") "
Oct 30 21:55:53.523371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.523267    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\" (UniqueName: \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\") pod \"02d09b60-e573-40c6-9d22-bdd339728f14\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") "
Oct 30 21:55:53.523371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.523320    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\") pod \"02d09b60-e573-40c6-9d22-bdd339728f14\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") "
Oct 30 21:55:53.523539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.523425    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\") pod \"02d09b60-e573-40c6-9d22-bdd339728f14\" (UID: \"02d09b60-e573-40c6-9d22-bdd339728f14\") "
Oct 30 21:55:53.523795 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.523677    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca" pod "02d09b60-e573-40c6-9d22-bdd339728f14" (UID: "02d09b60-e573-40c6-9d22-bdd339728f14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:53.524093 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.524069    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d09b60-e573-40c6-9d22-bdd339728f14-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:53.526074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.526047    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t" pod "02d09b60-e573-40c6-9d22-bdd339728f14" (UID: "02d09b60-e573-40c6-9d22-bdd339728f14"). InnerVolumeSpecName "kube-api-access-j9b9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:53.526466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.526442    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs" pod "02d09b60-e573-40c6-9d22-bdd339728f14" (UID: "02d09b60-e573-40c6-9d22-bdd339728f14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:53.528821 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.528796    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls" pod "02d09b60-e573-40c6-9d22-bdd339728f14" (UID: "02d09b60-e573-40c6-9d22-bdd339728f14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:53.636432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.626353    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9b9t\" (UniqueName: \"kubernetes.io/projected/02d09b60-e573-40c6-9d22-bdd339728f14-kube-api-access-j9b9t\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:53.636432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.626393    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:53.636432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.626429    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d09b60-e573-40c6-9d22-bdd339728f14-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:53.752101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.752010    7037 generic.go:356] "Generic (PLEG): container finished" podID="02d09b60-e573-40c6-9d22-bdd339728f14" containerID="9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6" exitCode=0
Oct 30 21:55:53.752101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.752070    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"
Oct 30 21:55:53.752631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.752093    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" event={"ID":"02d09b60-e573-40c6-9d22-bdd339728f14","Type":"ContainerDied","Data":"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"}
Oct 30 21:55:53.752631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.752223    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6" event={"ID":"02d09b60-e573-40c6-9d22-bdd339728f14","Type":"ContainerDied","Data":"27a274b582b2e7d7eda019a87ad3db90700fbc2f76b04024ab464d6559aa7433"}
Oct 30 21:55:53.752631 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.752263    7037 scope.go:122] "RemoveContainer" containerID="9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"
Oct 30 21:55:53.764978 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.764955    7037 scope.go:122] "RemoveContainer" containerID="9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"
Oct 30 21:55:53.765279 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:53.765257    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6\": container with ID starting with 9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6 not found: ID does not exist" containerID="9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"
Oct 30 21:55:53.765330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.765291    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6"} err="failed to get container status \"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6\": rpc error: code = NotFound desc = could not find container \"9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6\": container with ID starting with 9419ee5056e2db9155a2861b0dcaef313a6963912258421855243ace9fefccd6 not found: ID does not exist"
Oct 30 21:55:53.823665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.823633    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:55:53.835794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:53.835766    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-889195-75cc67f644-6mtr6"]
Oct 30 21:55:54.708068 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:54.708025    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d09b60-e573-40c6-9d22-bdd339728f14" path="/var/lib/kubelet/pods/02d09b60-e573-40c6-9d22-bdd339728f14/volumes"
Oct 30 21:55:57.351305 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:57.351263    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:58.329109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.329081    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:55:58.392009 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.391968    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\") pod \"df72b143-ed3f-4652-9c1d-cf400e8458a9\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") "
Oct 30 21:55:58.392485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.392088    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\" (UniqueName: \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\") pod \"df72b143-ed3f-4652-9c1d-cf400e8458a9\" (UID: \"df72b143-ed3f-4652-9c1d-cf400e8458a9\") "
Oct 30 21:55:58.392722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.392699    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume" pod "df72b143-ed3f-4652-9c1d-cf400e8458a9" (UID: "df72b143-ed3f-4652-9c1d-cf400e8458a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:58.394318 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.394293    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr" pod "df72b143-ed3f-4652-9c1d-cf400e8458a9" (UID: "df72b143-ed3f-4652-9c1d-cf400e8458a9"). InnerVolumeSpecName "kube-api-access-hxvtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:58.494111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.494078    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df72b143-ed3f-4652-9c1d-cf400e8458a9-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:58.494111 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.494105    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvtr\" (UniqueName: \"kubernetes.io/projected/df72b143-ed3f-4652-9c1d-cf400e8458a9-kube-api-access-hxvtr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:55:58.791048 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.791017    7037 generic.go:356] "Generic (PLEG): container finished" podID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerID="ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17" exitCode=0
Oct 30 21:55:58.791217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.791059    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"
Oct 30 21:55:58.791217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.791082    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" event={"ID":"df72b143-ed3f-4652-9c1d-cf400e8458a9","Type":"ContainerDied","Data":"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"}
Oct 30 21:55:58.791217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.791140    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn" event={"ID":"df72b143-ed3f-4652-9c1d-cf400e8458a9","Type":"ContainerDied","Data":"1eb8cf7045a03c5992f5c63cce2c065a3f588c2592e01826dcf490cc1ad9ffb5"}
Oct 30 21:55:58.791217 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.791171    7037 scope.go:122] "RemoveContainer" containerID="ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"
Oct 30 21:55:58.801342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.801313    7037 scope.go:122] "RemoveContainer" containerID="ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"
Oct 30 21:55:58.801682 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:55:58.801661    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17\": container with ID starting with ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17 not found: ID does not exist" containerID="ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"
Oct 30 21:55:58.801761 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.801691    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17"} err="failed to get container status \"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17\": rpc error: code = NotFound desc = could not find container \"ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17\": container with ID starting with ab0e38d7b8f3ecfa0c061c5a114bbe4d6971dfc54acd2b9b5677b7e5d04c1d17 not found: ID does not exist"
Oct 30 21:55:58.850326 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.850288    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:55:58.862001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:55:58.861972    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-889195-5dcfc78767-sldjn"]
Oct 30 21:56:00.709691 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:00.709653    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" path="/var/lib/kubelet/pods/df72b143-ed3f-4652-9c1d-cf400e8458a9/volumes"
Oct 30 21:56:01.863878 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:01.863840    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:56:01.879986 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:01.879953    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:56:01.910131 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:01.910100    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:56:03.673001 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:03.672965    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:56:07.776253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:07.776219    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:56:08.653328 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:08.653296    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:56:08.653544 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:08.653517    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" containerID="cri-o://bab208a68c088b8fb7f6122302fd2b9245fee6e3f9f3c174359c303eb04e223f" gracePeriod=30
Oct 30 21:56:09.256021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:09.255982    7037 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:09.256480 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:09.256050    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:12.777855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:12.777767    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing" containerID="cri-o://2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04" gracePeriod=25
Oct 30 21:56:13.358528 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.358483    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:56:13.358788 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.358761    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerName="maas-api" containerID="cri-o://684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4" gracePeriod=30
Oct 30 21:56:13.439859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.439815    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.1:8443/health\": dial tcp 10.134.1.1:8443: connect: connection refused"
Oct 30 21:56:13.659573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.659545    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:56:13.797323 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.797292    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") pod \"e0e3def0-0bfc-4501-95c7-bafc91844aef\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") "
Oct 30 21:56:13.797730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.797398    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\") pod \"e0e3def0-0bfc-4501-95c7-bafc91844aef\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") "
Oct 30 21:56:13.797730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.797562    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\") pod \"e0e3def0-0bfc-4501-95c7-bafc91844aef\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") "
Oct 30 21:56:13.797730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.797638    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\" (UniqueName: \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\") pod \"e0e3def0-0bfc-4501-95c7-bafc91844aef\" (UID: \"e0e3def0-0bfc-4501-95c7-bafc91844aef\") "
Oct 30 21:56:13.797940 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.797915    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca" pod "e0e3def0-0bfc-4501-95c7-bafc91844aef" (UID: "e0e3def0-0bfc-4501-95c7-bafc91844aef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:13.798150 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.798130    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e3def0-0bfc-4501-95c7-bafc91844aef-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:13.799996 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.799970    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls" pod "e0e3def0-0bfc-4501-95c7-bafc91844aef" (UID: "e0e3def0-0bfc-4501-95c7-bafc91844aef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:13.800101 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.800036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj" pod "e0e3def0-0bfc-4501-95c7-bafc91844aef" (UID: "e0e3def0-0bfc-4501-95c7-bafc91844aef"). InnerVolumeSpecName "kube-api-access-p46xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:13.800163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.800148    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs" pod "e0e3def0-0bfc-4501-95c7-bafc91844aef" (UID: "e0e3def0-0bfc-4501-95c7-bafc91844aef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:13.900293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.900198    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:13.900293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.900238    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p46xj\" (UniqueName: \"kubernetes.io/projected/e0e3def0-0bfc-4501-95c7-bafc91844aef-kube-api-access-p46xj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:13.900293 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.900253    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e3def0-0bfc-4501-95c7-bafc91844aef-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:13.914345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.913549    7037 generic.go:356] "Generic (PLEG): container finished" podID="b59227f7-56e1-4111-982c-2d1c3713368b" containerID="bab208a68c088b8fb7f6122302fd2b9245fee6e3f9f3c174359c303eb04e223f" exitCode=0
Oct 30 21:56:13.914345 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.913883    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" event={"ID":"b59227f7-56e1-4111-982c-2d1c3713368b","Type":"ContainerDied","Data":"bab208a68c088b8fb7f6122302fd2b9245fee6e3f9f3c174359c303eb04e223f"}
Oct 30 21:56:13.916934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.916368    7037 generic.go:356] "Generic (PLEG): container finished" podID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerID="684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4" exitCode=0
Oct 30 21:56:13.916934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.916466    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" event={"ID":"e0e3def0-0bfc-4501-95c7-bafc91844aef","Type":"ContainerDied","Data":"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"}
Oct 30 21:56:13.916934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.916527    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk" event={"ID":"e0e3def0-0bfc-4501-95c7-bafc91844aef","Type":"ContainerDied","Data":"310cd6dae521138e7354b3f81a91a7e971a7935712c2eb3e7bb4a369751994a4"}
Oct 30 21:56:13.916934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.916560    7037 scope.go:122] "RemoveContainer" containerID="684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"
Oct 30 21:56:13.916934 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.916756    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"
Oct 30 21:56:13.965530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.963578    7037 scope.go:122] "RemoveContainer" containerID="684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"
Oct 30 21:56:13.965530 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:13.964664    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4\": container with ID starting with 684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4 not found: ID does not exist" containerID="684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"
Oct 30 21:56:13.965530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.964708    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4"} err="failed to get container status \"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4\": rpc error: code = NotFound desc = could not find container \"684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4\": container with ID starting with 684ec195bdff9a42801f8889026a61dec20e85ccc1512c1c7bccfad56908bae4 not found: ID does not exist"
Oct 30 21:56:13.978156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.978127    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:56:13.998723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:13.998693    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:56:14.004321 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.004296    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8137c-8b4d56cd7-llllk"]
Oct 30 21:56:14.103061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103029    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103121    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103183    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103235    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103330 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103297    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103441 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103338    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103441 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103434    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103532    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.103629 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103585    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\") pod \"b59227f7-56e1-4111-982c-2d1c3713368b\" (UID: \"b59227f7-56e1-4111-982c-2d1c3713368b\") "
Oct 30 21:56:14.104015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103477    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:14.104015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.103979    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:14.104241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.104194    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.104241 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.104216    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.104370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.104240    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:14.104370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.104315    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:14.104607 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.104583    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:14.105561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.105525    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "kube-api-access-jv96f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:14.105705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.105688    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:14.105891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.105869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:14.105938 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.105923    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy" pod "b59227f7-56e1-4111-982c-2d1c3713368b" (UID: "b59227f7-56e1-4111-982c-2d1c3713368b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205671    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59227f7-56e1-4111-982c-2d1c3713368b-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205702    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205751    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv96f\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-kube-api-access-jv96f\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205776    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59227f7-56e1-4111-982c-2d1c3713368b-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205813    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205828    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59227f7-56e1-4111-982c-2d1c3713368b-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.205895 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.205840    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59227f7-56e1-4111-982c-2d1c3713368b-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:14.708433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.708396    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" path="/var/lib/kubelet/pods/e0e3def0-0bfc-4501-95c7-bafc91844aef/volumes"
Oct 30 21:56:14.935779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.935741    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f" event={"ID":"b59227f7-56e1-4111-982c-2d1c3713368b","Type":"ContainerDied","Data":"fb8c4b5eff7b91b69cfd68035b13a1c8eb68ced6aeb3a7c9ec946f4d9430600b"}
Oct 30 21:56:14.935779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.935764    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"
Oct 30 21:56:14.936341 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.935825    7037 scope.go:122] "RemoveContainer" containerID="bab208a68c088b8fb7f6122302fd2b9245fee6e3f9f3c174359c303eb04e223f"
Oct 30 21:56:14.995329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:14.995255    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:56:15.007552 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:15.007519    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-m595f"]
Oct 30 21:56:15.299224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:15.299132    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:15.947916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:15.947881    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:56:16.709361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:16.709329    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" path="/var/lib/kubelet/pods/b59227f7-56e1-4111-982c-2d1c3713368b/volumes"
Oct 30 21:56:16.886041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:16.886001    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main" containerID="cri-o://25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e" gracePeriod=45
Oct 30 21:56:16.900798 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:16.900764    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" containerID="cri-o://1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac" gracePeriod=45
Oct 30 21:56:16.931609 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:16.931551    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main" containerID="cri-o://d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62" gracePeriod=45
Oct 30 21:56:17.071253 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.071202    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 30 21:56:17.234344 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.234280    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:56:17.257133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.257105    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:56:17.260925 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.260895    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:56:17.353467 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353431    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.353689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353500    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.353689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353549    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.353689 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353646    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353727    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\" (UniqueName: \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353756    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353766    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353789    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353830    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.353891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353869    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353900    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353947    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.353965    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354033    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\") pod \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\" (UID: \"141a6dca-9c9e-44eb-9b63-58bf541f80ba\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354078    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354110    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354143    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354149    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354176    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\" (UniqueName: \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354237    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\" (UniqueName: \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\") pod \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\" (UID: \"6e1d52d9-d0cd-4322-969b-3a3e841db30f\") "
Oct 30 21:56:17.354309 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354274    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\") pod \"ad803c3d-e2e8-400f-995b-c81cd7d60772\" (UID: \"ad803c3d-e2e8-400f-995b-c81cd7d60772\") "
Oct 30 21:56:17.354899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354739    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.354960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354932    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.354960 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354955    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.355076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354969    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.355076 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.354983    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.355688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.355660    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.357325 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.357296    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.357992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.357951    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.357992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.357968    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.358172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.357954    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.359272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.359229    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "kube-api-access-7b8v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.360556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.360482    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365474    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365508    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "kube-api-access-bnhrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365616    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365684    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs" pod "6e1d52d9-d0cd-4322-969b-3a3e841db30f" (UID: "6e1d52d9-d0cd-4322-969b-3a3e841db30f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365697    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "kube-api-access-99hvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.365819 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.365727    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm" pod "ad803c3d-e2e8-400f-995b-c81cd7d60772" (UID: "ad803c3d-e2e8-400f-995b-c81cd7d60772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.367095 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.367071    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm" pod "141a6dca-9c9e-44eb-9b63-58bf541f80ba" (UID: "141a6dca-9c9e-44eb-9b63-58bf541f80ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:17.456021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.455984    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8v4\" (UniqueName: \"kubernetes.io/projected/141a6dca-9c9e-44eb-9b63-58bf541f80ba-kube-api-access-7b8v4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456249    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456282    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456296    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456310    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456322    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456334    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141a6dca-9c9e-44eb-9b63-58bf541f80ba-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456346    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456358    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad803c3d-e2e8-400f-995b-c81cd7d60772-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456370    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e1d52d9-d0cd-4322-969b-3a3e841db30f-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456382    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hvz\" (UniqueName: \"kubernetes.io/projected/ad803c3d-e2e8-400f-995b-c81cd7d60772-kube-api-access-99hvz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456395    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhrw\" (UniqueName: \"kubernetes.io/projected/6e1d52d9-d0cd-4322-969b-3a3e841db30f-kube-api-access-bnhrw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456410    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad803c3d-e2e8-400f-995b-c81cd7d60772-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.456557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.456424    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1d52d9-d0cd-4322-969b-3a3e841db30f-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:17.960485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.960452    7037 generic.go:356] "Generic (PLEG): container finished" podID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerID="d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62" exitCode=0
Oct 30 21:56:17.960673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.960535    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"
Oct 30 21:56:17.960673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.960545    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" event={"ID":"ad803c3d-e2e8-400f-995b-c81cd7d60772","Type":"ContainerDied","Data":"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"}
Oct 30 21:56:17.960673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.960595    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc" event={"ID":"ad803c3d-e2e8-400f-995b-c81cd7d60772","Type":"ContainerDied","Data":"70204344e7cb23a572dec23fc8c97a44afd2e7c6e17ed4646f1637a6248cfd22"}
Oct 30 21:56:17.960673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.960619    7037 scope.go:122] "RemoveContainer" containerID="d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"
Oct 30 21:56:17.962368 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.962345    7037 generic.go:356] "Generic (PLEG): container finished" podID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerID="25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e" exitCode=0
Oct 30 21:56:17.962516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.962384    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"
Oct 30 21:56:17.962516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.962423    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" event={"ID":"6e1d52d9-d0cd-4322-969b-3a3e841db30f","Type":"ContainerDied","Data":"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"}
Oct 30 21:56:17.962516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.962461    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt" event={"ID":"6e1d52d9-d0cd-4322-969b-3a3e841db30f","Type":"ContainerDied","Data":"7a71b1f0648331989182fe2efde1d26732f8c5a3f77dbb0575b4bdd6055726d1"}
Oct 30 21:56:17.963948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.963916    7037 generic.go:356] "Generic (PLEG): container finished" podID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerID="1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac" exitCode=0
Oct 30 21:56:17.964057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.963961    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"
Oct 30 21:56:17.964057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.963959    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" event={"ID":"141a6dca-9c9e-44eb-9b63-58bf541f80ba","Type":"ContainerDied","Data":"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"}
Oct 30 21:56:17.964177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.964073    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb" event={"ID":"141a6dca-9c9e-44eb-9b63-58bf541f80ba","Type":"ContainerDied","Data":"28981e0087da4650a25354fc7f5d69a8bb333a35b3a550905333b899f299d594"}
Oct 30 21:56:17.972909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.972888    7037 scope.go:122] "RemoveContainer" containerID="d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"
Oct 30 21:56:17.973177 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:17.973160    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62\": container with ID starting with d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62 not found: ID does not exist" containerID="d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"
Oct 30 21:56:17.973243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.973183    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62"} err="failed to get container status \"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62\": rpc error: code = NotFound desc = could not find container \"d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62\": container with ID starting with d12196acabd92483e2f6b985acadb6e66d61b068969eba8249d3f2c1308a5e62 not found: ID does not exist"
Oct 30 21:56:17.973243 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.973200    7037 scope.go:122] "RemoveContainer" containerID="25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"
Oct 30 21:56:17.985249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.985235    7037 scope.go:122] "RemoveContainer" containerID="25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"
Oct 30 21:56:17.985544 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:17.985525    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e\": container with ID starting with 25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e not found: ID does not exist" containerID="25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"
Oct 30 21:56:17.985612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.985550    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e"} err="failed to get container status \"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e\": rpc error: code = NotFound desc = could not find container \"25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e\": container with ID starting with 25a29b007b2ba086364a7ec77ab8673a98262ca5473684170e1a59babd829f8e not found: ID does not exist"
Oct 30 21:56:17.985612 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.985565    7037 scope.go:122] "RemoveContainer" containerID="1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"
Oct 30 21:56:17.994409 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.994388    7037 scope.go:122] "RemoveContainer" containerID="1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"
Oct 30 21:56:17.994685 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:17.994660    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac\": container with ID starting with 1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac not found: ID does not exist" containerID="1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"
Oct 30 21:56:17.994764 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:17.994694    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac"} err="failed to get container status \"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac\": rpc error: code = NotFound desc = could not find container \"1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac\": container with ID starting with 1b3b9a183cbde18631f540efa247a7a8fa8df15b14fcd96869b7f294c99cccac not found: ID does not exist"
Oct 30 21:56:18.029461 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.029435    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:56:18.033230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.033200    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/simulator-w0-58fea9-kserve-7f5d684569-w4kpc"]
Oct 30 21:56:18.082805 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.082779    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:56:18.086929 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.086907    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-2-w0-58fea9-kserve-6f447fb9bd-snwdt"]
Oct 30 21:56:18.144335 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.144305    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:56:18.147484 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.147466    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-58fea9/distinct-w0-58fea9-kserve-77d44f99d7-xqsnb"]
Oct 30 21:56:18.710726 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.710693    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" path="/var/lib/kubelet/pods/141a6dca-9c9e-44eb-9b63-58bf541f80ba/volumes"
Oct 30 21:56:18.711324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.711307    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" path="/var/lib/kubelet/pods/6e1d52d9-d0cd-4322-969b-3a3e841db30f/volumes"
Oct 30 21:56:18.711760 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:18.711745    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" path="/var/lib/kubelet/pods/ad803c3d-e2e8-400f-995b-c81cd7d60772/volumes"
Oct 30 21:56:20.948563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:20.948514    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" podUID="e7716237-5b0f-43aa-824f-1a74121d9a29" containerName="payload-processing" containerID="cri-o://0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf" gracePeriod=25
Oct 30 21:56:21.249630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.249601    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:56:21.300408 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.300377    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\" (UniqueName: \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\") pod \"e7716237-5b0f-43aa-824f-1a74121d9a29\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") "
Oct 30 21:56:21.300600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.300460    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\") pod \"e7716237-5b0f-43aa-824f-1a74121d9a29\" (UID: \"e7716237-5b0f-43aa-824f-1a74121d9a29\") "
Oct 30 21:56:21.300865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.300827    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume" pod "e7716237-5b0f-43aa-824f-1a74121d9a29" (UID: "e7716237-5b0f-43aa-824f-1a74121d9a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:21.301005 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.300973    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7716237-5b0f-43aa-824f-1a74121d9a29-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:21.302573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.302546    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh" pod "e7716237-5b0f-43aa-824f-1a74121d9a29" (UID: "e7716237-5b0f-43aa-824f-1a74121d9a29"). InnerVolumeSpecName "kube-api-access-tmdfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:21.410329 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:21.410287    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdfh\" (UniqueName: \"kubernetes.io/projected/e7716237-5b0f-43aa-824f-1a74121d9a29-kube-api-access-tmdfh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:22.010677 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.010641    7037 generic.go:356] "Generic (PLEG): container finished" podID="e7716237-5b0f-43aa-824f-1a74121d9a29" containerID="0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf" exitCode=0
Oct 30 21:56:22.011200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.010709    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"
Oct 30 21:56:22.011200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.010735    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" event={"ID":"e7716237-5b0f-43aa-824f-1a74121d9a29","Type":"ContainerDied","Data":"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"}
Oct 30 21:56:22.011200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.010820    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc" event={"ID":"e7716237-5b0f-43aa-824f-1a74121d9a29","Type":"ContainerDied","Data":"a1cc138579f247ec2b02a29dd9ff506fac9e6921b042e6f2107bbb82956fea1a"}
Oct 30 21:56:22.011200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.010871    7037 scope.go:122] "RemoveContainer" containerID="0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"
Oct 30 21:56:22.025942 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.025924    7037 scope.go:122] "RemoveContainer" containerID="0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"
Oct 30 21:56:22.026197 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:22.026177    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf\": container with ID starting with 0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf not found: ID does not exist" containerID="0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"
Oct 30 21:56:22.026246 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.026203    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf"} err="failed to get container status \"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf\": rpc error: code = NotFound desc = could not find container \"0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf\": container with ID starting with 0a3b1078d8abc76295726f3cda687cb08231dff6aed99bc74abf8b625884addf not found: ID does not exist"
Oct 30 21:56:22.074322 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.074294    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:56:22.085853 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.085817    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-56ac74-96bdd884-tp8pc"]
Oct 30 21:56:22.707308 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:22.707272    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7716237-5b0f-43aa-824f-1a74121d9a29" path="/var/lib/kubelet/pods/e7716237-5b0f-43aa-824f-1a74121d9a29/volumes"
Oct 30 21:56:25.299668 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:25.299628    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:33.075605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.075578    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:56:33.078248 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.078215    7037 generic.go:356] "Generic (PLEG): container finished" podID="25d9b503-6346-439a-81be-4fb943579121" containerID="2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04" exitCode=0
Oct 30 21:56:33.078384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.078261    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"
Oct 30 21:56:33.078384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.078296    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" event={"ID":"25d9b503-6346-439a-81be-4fb943579121","Type":"ContainerDied","Data":"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"}
Oct 30 21:56:33.078384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.078363    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87" event={"ID":"25d9b503-6346-439a-81be-4fb943579121","Type":"ContainerDied","Data":"d10947e1888c650d97e7d823fbb9b530bff3ed9413fbba01e7f6d100350b7125"}
Oct 30 21:56:33.078384 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.078378    7037 scope.go:122] "RemoveContainer" containerID="2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"
Oct 30 21:56:33.088641 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.088611    7037 scope.go:122] "RemoveContainer" containerID="2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"
Oct 30 21:56:33.089020 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:33.088935    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04\": container with ID starting with 2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04 not found: ID does not exist" containerID="2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"
Oct 30 21:56:33.089141 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.089072    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04"} err="failed to get container status \"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04\": rpc error: code = NotFound desc = could not find container \"2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04\": container with ID starting with 2ab5b9a6e183c4673681f5a35daf03542b40e50034587b478b88189e75259c04 not found: ID does not exist"
Oct 30 21:56:33.186139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.186091    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\") pod \"25d9b503-6346-439a-81be-4fb943579121\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") "
Oct 30 21:56:33.186342 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.186249    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\" (UniqueName: \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\") pod \"25d9b503-6346-439a-81be-4fb943579121\" (UID: \"25d9b503-6346-439a-81be-4fb943579121\") "
Oct 30 21:56:33.187516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.187471    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume" pod "25d9b503-6346-439a-81be-4fb943579121" (UID: "25d9b503-6346-439a-81be-4fb943579121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:33.191081 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.190982    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb" pod "25d9b503-6346-439a-81be-4fb943579121" (UID: "25d9b503-6346-439a-81be-4fb943579121"). InnerVolumeSpecName "kube-api-access-kkfhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:33.288420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.288382    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfhb\" (UniqueName: \"kubernetes.io/projected/25d9b503-6346-439a-81be-4fb943579121-kube-api-access-kkfhb\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:33.288420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.288422    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25d9b503-6346-439a-81be-4fb943579121-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:33.431123 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.430040    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:56:33.440292 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:33.440261    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-58fea9-9984455b6-brp87"]
Oct 30 21:56:33.677072 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:33.676985    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:56:34.709358 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:34.709317    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d9b503-6346-439a-81be-4fb943579121" path="/var/lib/kubelet/pods/25d9b503-6346-439a-81be-4fb943579121/volumes"
Oct 30 21:56:35.818436 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:35.818343    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:56:40.819401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:40.819331    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing" containerID="cri-o://25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9" gracePeriod=25
Oct 30 21:56:48.060435 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.060401    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:56:48.061028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.060646    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" podUID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" containerName="maas-api" containerID="cri-o://fe33c1a803b3df522a9f0bea26fc6c3276370b14112ea682373d366db9c9e87a" gracePeriod=30
Oct 30 21:56:48.178502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.178456    7037 generic.go:356] "Generic (PLEG): container finished" podID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" containerID="fe33c1a803b3df522a9f0bea26fc6c3276370b14112ea682373d366db9c9e87a" exitCode=0
Oct 30 21:56:48.178700 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.178529    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" event={"ID":"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0","Type":"ContainerDied","Data":"fe33c1a803b3df522a9f0bea26fc6c3276370b14112ea682373d366db9c9e87a"}
Oct 30 21:56:48.364561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.364533    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:56:48.497732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.497703    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\") pod \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") "
Oct 30 21:56:48.497902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.497762    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\" (UniqueName: \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\") pod \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") "
Oct 30 21:56:48.497902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.497785    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\") pod \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") "
Oct 30 21:56:48.497902 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.497839    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\") pod \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\" (UID: \"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0\") "
Oct 30 21:56:48.498070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.498047    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca" pod "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" (UID: "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:48.498395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.498362    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:48.499988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.499958    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls" pod "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" (UID: "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:48.500117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.499984    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs" pod "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" (UID: "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:48.500117 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.500032    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn" pod "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" (UID: "b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0"). InnerVolumeSpecName "kube-api-access-mw6vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:48.599389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.599301    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6vn\" (UniqueName: \"kubernetes.io/projected/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-kube-api-access-mw6vn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:48.599389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.599336    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:48.599389 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.599345    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:48.822595 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:48.822550    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:49.186582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:49.186547    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"
Oct 30 21:56:49.186582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:49.186543    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg" event={"ID":"b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0","Type":"ContainerDied","Data":"e11285174fa7856d703ce3b569d0550b307a1507f8dda33a09100654cd189bbb"}
Oct 30 21:56:49.187103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:49.186627    7037 scope.go:122] "RemoveContainer" containerID="fe33c1a803b3df522a9f0bea26fc6c3276370b14112ea682373d366db9c9e87a"
Oct 30 21:56:49.255327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:49.255287    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:56:49.279886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:49.279854    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c46397-77bf85bf5c-zn4pg"]
Oct 30 21:56:50.708119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:50.708088    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" path="/var/lib/kubelet/pods/b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0/volumes"
Oct 30 21:56:51.147825 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:51.147790    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:56:51.148056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:51.148024    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" containerID="cri-o://d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b" gracePeriod=30
Oct 30 21:56:54.856951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:54.856918    7037 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:54.857359 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:54.856974    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:56.418956 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.418882    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:56:56.502405 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502366    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502413    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502466    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502552    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502623 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502590    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502645    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502732    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502751    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502764    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502805    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502836    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\") pod \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\" (UID: \"f70fdb94-df8e-4e9c-aa5d-1c924113345d\") "
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502851    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:56.502880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.502867    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:56.503311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.503262    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:56.504250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.504223    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.504250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.504253    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.504454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.504268    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.504454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.504285    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.504454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.504302    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.505530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.505223    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:56.505530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.505433    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:56.505530 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.505441    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "kube-api-access-5gm57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:56.505776 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.505646    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy" pod "f70fdb94-df8e-4e9c-aa5d-1c924113345d" (UID: "f70fdb94-df8e-4e9c-aa5d-1c924113345d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:56.605725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.605695    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.605725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.605721    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.605725 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.605730    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70fdb94-df8e-4e9c-aa5d-1c924113345d-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:56.605973 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:56.605739    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gm57\" (UniqueName: \"kubernetes.io/projected/f70fdb94-df8e-4e9c-aa5d-1c924113345d-kube-api-access-5gm57\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:56:57.060315 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.060240    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:56:57.060532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.060473    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy" containerID="cri-o://22779482cbffc757ff315e6bbe4543dc7a8e17fee21e396b16b33a37ccab5a30" gracePeriod=30
Oct 30 21:56:57.239502 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.239464    7037 generic.go:356] "Generic (PLEG): container finished" podID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerID="d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b" exitCode=0
Oct 30 21:56:57.239682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.239537    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"
Oct 30 21:56:57.239682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.239554    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" event={"ID":"f70fdb94-df8e-4e9c-aa5d-1c924113345d","Type":"ContainerDied","Data":"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"}
Oct 30 21:56:57.239682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.239611    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj" event={"ID":"f70fdb94-df8e-4e9c-aa5d-1c924113345d","Type":"ContainerDied","Data":"dfd1f21c9ae60012d5cd89575de4bb33bda966fa49c28c1666a92c3867553da0"}
Oct 30 21:56:57.239682 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.239634    7037 scope.go:122] "RemoveContainer" containerID="d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"
Oct 30 21:56:57.250099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.250078    7037 scope.go:122] "RemoveContainer" containerID="d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"
Oct 30 21:56:57.250365 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:56:57.250345    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b\": container with ID starting with d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b not found: ID does not exist" containerID="d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"
Oct 30 21:56:57.250418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.250373    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b"} err="failed to get container status \"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b\": rpc error: code = NotFound desc = could not find container \"d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b\": container with ID starting with d28d7b2bbed57a264629b06879b13fb957ea4f10bb50f1ec56d0697f5e8e8c2b not found: ID does not exist"
Oct 30 21:56:57.333290 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.333262    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:56:57.344128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:57.344101    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-zmzrj"]
Oct 30 21:56:58.707770 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:58.707737    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" path="/var/lib/kubelet/pods/f70fdb94-df8e-4e9c-aa5d-1c924113345d/volumes"
Oct 30 21:56:58.823146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:56:58.823081    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:01.098855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.098825    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:57:01.159175 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.159143    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\" (UniqueName: \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\") pod \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") "
Oct 30 21:57:01.159343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.159256    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\") pod \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\" (UID: \"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83\") "
Oct 30 21:57:01.159756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.159726    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume" pod "2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" (UID: "2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:01.161385 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.161361    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll" pod "2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" (UID: "2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83"). InnerVolumeSpecName "kube-api-access-knwll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:01.261582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.261545    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:01.261582 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.261581    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwll\" (UniqueName: \"kubernetes.io/projected/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83-kube-api-access-knwll\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:01.266006 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.265978    7037 generic.go:356] "Generic (PLEG): container finished" podID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerID="25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9" exitCode=0
Oct 30 21:57:01.266119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.266031    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"
Oct 30 21:57:01.266119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.266054    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" event={"ID":"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83","Type":"ContainerDied","Data":"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"}
Oct 30 21:57:01.266119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.266092    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq" event={"ID":"2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83","Type":"ContainerDied","Data":"9315e738ab56e006ab2ab2e993936a2f308c289e225ee6a0bae70892ee55b46d"}
Oct 30 21:57:01.266119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.266112    7037 scope.go:122] "RemoveContainer" containerID="25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"
Oct 30 21:57:01.276172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.276150    7037 scope.go:122] "RemoveContainer" containerID="25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"
Oct 30 21:57:01.276437 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:01.276416    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9\": container with ID starting with 25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9 not found: ID does not exist" containerID="25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"
Oct 30 21:57:01.276546 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.276443    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9"} err="failed to get container status \"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9\": rpc error: code = NotFound desc = could not find container \"25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9\": container with ID starting with 25570f85be5b3fd609620001a3008b524ce4afb847ffebf2af784680cc0ea9c9 not found: ID does not exist"
Oct 30 21:57:01.332463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.332384    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:57:01.344247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:01.344213    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-55c5e0-849756dbfb-wfvzq"]
Oct 30 21:57:02.276905 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.276872    7037 generic.go:356] "Generic (PLEG): container finished" podID="bf3db193-1d90-49e9-9013-59bcbf381888" containerID="22779482cbffc757ff315e6bbe4543dc7a8e17fee21e396b16b33a37ccab5a30" exitCode=0
Oct 30 21:57:02.277369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.276953    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerDied","Data":"22779482cbffc757ff315e6bbe4543dc7a8e17fee21e396b16b33a37ccab5a30"}
Oct 30 21:57:02.277369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.277069    7037 scope.go:122] "RemoveContainer" containerID="25406431f06b088d1ac0c485331a91be4aea9a82457b145c3c81fb3ddda76410"
Oct 30 21:57:02.329034 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.328976    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:57:02.376413 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376386    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376428    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376464    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376652 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376610    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376701 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376661    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376755 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376698    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.376811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376751    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:02.377200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376922    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:02.377200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376967    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.377200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.376995    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:02.377200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.377118    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.377200 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.377196    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\") pod \"bf3db193-1d90-49e9-9013-59bcbf381888\" (UID: \"bf3db193-1d90-49e9-9013-59bcbf381888\") "
Oct 30 21:57:02.377880 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.377808    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:02.378447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.378396    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:02.378597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.378541    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.378597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.378568    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.378597 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.378584    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.378794 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.378600    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.380160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.380090    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "kube-api-access-m9dcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:02.380273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.380205    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:02.380742 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.380712    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:02.381949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.381926    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy" pod "bf3db193-1d90-49e9-9013-59bcbf381888" (UID: "bf3db193-1d90-49e9-9013-59bcbf381888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:02.479756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.479722    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.479756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.479752    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf3db193-1d90-49e9-9013-59bcbf381888-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.479756 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.479763    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dcr\" (UniqueName: \"kubernetes.io/projected/bf3db193-1d90-49e9-9013-59bcbf381888-kube-api-access-m9dcr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.479993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.479773    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf3db193-1d90-49e9-9013-59bcbf381888-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.479993 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.479782    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf3db193-1d90-49e9-9013-59bcbf381888-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:02.706894 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:02.706809    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" path="/var/lib/kubelet/pods/2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83/volumes"
Oct 30 21:57:03.283780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:03.283748    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"
Oct 30 21:57:03.284278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:03.283793    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4" event={"ID":"bf3db193-1d90-49e9-9013-59bcbf381888","Type":"ContainerDied","Data":"661fbd7a3d2d962083be88f63d60cc59b7a9da1fd9e359c7ab744f6e9cacb9e7"}
Oct 30 21:57:03.284278 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:03.283858    7037 scope.go:122] "RemoveContainer" containerID="22779482cbffc757ff315e6bbe4543dc7a8e17fee21e396b16b33a37ccab5a30"
Oct 30 21:57:03.339501 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:03.339459    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:57:03.351849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:03.351816    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-ss9t4"]
Oct 30 21:57:03.672262 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:03.672227    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:57:04.211361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.211328    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:57:04.211590 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.211559    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" podUID="c9c7766e-85f0-47ef-a75f-dd2e064114db" containerName="maas-api" containerID="cri-o://2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f" gracePeriod=30
Oct 30 21:57:04.488723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.488695    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:57:04.603883 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.603852    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\") pod \"c9c7766e-85f0-47ef-a75f-dd2e064114db\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") "
Oct 30 21:57:04.604070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.603910    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\") pod \"c9c7766e-85f0-47ef-a75f-dd2e064114db\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") "
Oct 30 21:57:04.604070 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.603983    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\") pod \"c9c7766e-85f0-47ef-a75f-dd2e064114db\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") "
Oct 30 21:57:04.604208 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.604098    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\" (UniqueName: \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\") pod \"c9c7766e-85f0-47ef-a75f-dd2e064114db\" (UID: \"c9c7766e-85f0-47ef-a75f-dd2e064114db\") "
Oct 30 21:57:04.604304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.604277    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca" pod "c9c7766e-85f0-47ef-a75f-dd2e064114db" (UID: "c9c7766e-85f0-47ef-a75f-dd2e064114db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.604727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.604663    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7766e-85f0-47ef-a75f-dd2e064114db-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:04.606159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.606131    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs" pod "c9c7766e-85f0-47ef-a75f-dd2e064114db" (UID: "c9c7766e-85f0-47ef-a75f-dd2e064114db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:04.606262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.606198    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx" pod "c9c7766e-85f0-47ef-a75f-dd2e064114db" (UID: "c9c7766e-85f0-47ef-a75f-dd2e064114db"). InnerVolumeSpecName "kube-api-access-qxczx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.606262 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.606207    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls" pod "c9c7766e-85f0-47ef-a75f-dd2e064114db" (UID: "c9c7766e-85f0-47ef-a75f-dd2e064114db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:04.705365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.705336    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:04.705365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.705363    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxczx\" (UniqueName: \"kubernetes.io/projected/c9c7766e-85f0-47ef-a75f-dd2e064114db-kube-api-access-qxczx\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:04.705365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.705372    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c7766e-85f0-47ef-a75f-dd2e064114db-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:04.707512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:04.707476    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" path="/var/lib/kubelet/pods/bf3db193-1d90-49e9-9013-59bcbf381888/volumes"
Oct 30 21:57:05.299927 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.299886    7037 generic.go:356] "Generic (PLEG): container finished" podID="c9c7766e-85f0-47ef-a75f-dd2e064114db" containerID="2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f" exitCode=0
Oct 30 21:57:05.300102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.299931    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"
Oct 30 21:57:05.300102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.299947    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" event={"ID":"c9c7766e-85f0-47ef-a75f-dd2e064114db","Type":"ContainerDied","Data":"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"}
Oct 30 21:57:05.300102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.300007    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl" event={"ID":"c9c7766e-85f0-47ef-a75f-dd2e064114db","Type":"ContainerDied","Data":"88dc9e501208854763503204e231e457f8daf8744d8b80eac3ca38a4d799469d"}
Oct 30 21:57:05.300102 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.300039    7037 scope.go:122] "RemoveContainer" containerID="2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"
Oct 30 21:57:05.310747 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.310721    7037 scope.go:122] "RemoveContainer" containerID="2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"
Oct 30 21:57:05.311102 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:05.311084    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f\": container with ID starting with 2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f not found: ID does not exist" containerID="2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"
Oct 30 21:57:05.311146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.311113    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f"} err="failed to get container status \"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f\": rpc error: code = NotFound desc = could not find container \"2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f\": container with ID starting with 2bb61285eab4c61584ce3ef9d2033c6a19576f8e13ed72b7f60e993454bd4c3f not found: ID does not exist"
Oct 30 21:57:05.353157 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.353119    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:57:05.360879 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:05.360854    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be3ada-76d7bb9754-9wkhl"]
Oct 30 21:57:06.490757 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:06.490721    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:57:06.503045 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:06.503013    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:57:06.708109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:06.708078    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c7766e-85f0-47ef-a75f-dd2e064114db" path="/var/lib/kubelet/pods/c9c7766e-85f0-47ef-a75f-dd2e064114db/volumes"
Oct 30 21:57:07.881719 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:07.881685    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:57:07.882146 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:07.881949    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy" containerID="cri-o://887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e" gracePeriod=30
Oct 30 21:57:13.156922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.156854    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:57:13.311196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311196 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311199    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311219    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311256    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311346    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311463 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311390    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311607    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311658    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311694    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:13.311727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311707    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:13.311727 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311700    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:13.311992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311745    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\") pod \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\" (UID: \"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d\") "
Oct 30 21:57:13.311992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.311892    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:13.312099 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312029    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:13.312250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312221    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.312250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312247    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.312416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312260    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.312416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312268    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.312416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.312277    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.313659 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.313631    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:13.313811 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.313790    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:13.314062 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.314040    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "kube-api-access-gzd8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:13.314125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.314070    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo" pod "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" (UID: "b5a0b1b2-b5bb-412c-87fc-5bc3492a709d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:13.355988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.355950    7037 generic.go:356] "Generic (PLEG): container finished" podID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerID="887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e" exitCode=0
Oct 30 21:57:13.356172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.355999    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"
Oct 30 21:57:13.356172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.356038    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" event={"ID":"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d","Type":"ContainerDied","Data":"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"}
Oct 30 21:57:13.356172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.356101    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z" event={"ID":"b5a0b1b2-b5bb-412c-87fc-5bc3492a709d","Type":"ContainerDied","Data":"4e6e78e15b0584fdb2cf9ea94ad6d88d2dbc1d25d80ffe370d1d023efad24065"}
Oct 30 21:57:13.356172 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.356130    7037 scope.go:122] "RemoveContainer" containerID="887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"
Oct 30 21:57:13.367837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.367813    7037 scope.go:122] "RemoveContainer" containerID="887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"
Oct 30 21:57:13.368130 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:13.368114    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e\": container with ID starting with 887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e not found: ID does not exist" containerID="887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"
Oct 30 21:57:13.368184 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.368139    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e"} err="failed to get container status \"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e\": rpc error: code = NotFound desc = could not find container \"887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e\": container with ID starting with 887b1272171f0b29e48795596d3b018e4c13a101abe14aeb702fc8ccd142c13e not found: ID does not exist"
Oct 30 21:57:13.413532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.413443    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.413532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.413473    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzd8q\" (UniqueName: \"kubernetes.io/projected/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-kube-api-access-gzd8q\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.413532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.413483    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.413532 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.413512    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:13.435632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.435601    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:57:13.454198 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:13.454166    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-zdv9z"]
Oct 30 21:57:14.707645 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:14.707611    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" path="/var/lib/kubelet/pods/b5a0b1b2-b5bb-412c-87fc-5bc3492a709d/volumes"
Oct 30 21:57:15.752477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:15.752444    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:57:15.822789 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:15.822755    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:57:20.293270 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:20.293234    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:57:20.293908 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:20.293471    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy" containerID="cri-o://daafcf3b5d02b5a63ffaae8168e14fe3ebf4fbfc67b5e8cf080fc088deaec309" gracePeriod=30
Oct 30 21:57:20.752913 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:20.752859    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing" containerID="cri-o://0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9" gracePeriod=25
Oct 30 21:57:20.823974 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:20.823933    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing" containerID="cri-o://3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c" gracePeriod=25
Oct 30 21:57:21.511785 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.511738    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main" containerID="cri-o://a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a" gracePeriod=45
Oct 30 21:57:21.531144 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.530936    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" containerID="cri-o://50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6" gracePeriod=45
Oct 30 21:57:21.801865 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.801838    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:57:21.824125 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.824095    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:57:21.908316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908278    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.908316 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908323    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.908630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908408    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.908630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908448    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.908630 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908524    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.908809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908629    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.908809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908647    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.908809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908721    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\" (UniqueName: \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.908809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908753    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\" (UniqueName: \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.908809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908780    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.909090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908816    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.909090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908824    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.909090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908868    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\") pod \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\" (UID: \"da9c767d-9290-4cde-87d3-0d3bb6ecef8c\") "
Oct 30 21:57:21.909090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.908896    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\") pod \"a0a7d957-d295-4ee4-9588-57819e924eeb\" (UID: \"a0a7d957-d295-4ee4-9588-57819e924eeb\") "
Oct 30 21:57:21.909090 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909056    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.909475 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909450    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:21.909596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909475    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:21.909596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909502    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:21.909596 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909451    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.909758 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909740    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.909923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.909897    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.911806 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.911783    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.914810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.914783    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "kube-api-access-lsqkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:21.914810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.914797    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:21.914810 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.914792    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:21.914988 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.914810    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm" pod "da9c767d-9290-4cde-87d3-0d3bb6ecef8c" (UID: "da9c767d-9290-4cde-87d3-0d3bb6ecef8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:21.915989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:21.915972    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk" pod "a0a7d957-d295-4ee4-9588-57819e924eeb" (UID: "a0a7d957-d295-4ee4-9588-57819e924eeb"). InnerVolumeSpecName "kube-api-access-227tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:22.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011000    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011041    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a7d957-d295-4ee4-9588-57819e924eeb-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011051    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227tk\" (UniqueName: \"kubernetes.io/projected/a0a7d957-d295-4ee4-9588-57819e924eeb-kube-api-access-227tk\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011061    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqkl\" (UniqueName: \"kubernetes.io/projected/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-kube-api-access-lsqkl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011069    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011077    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011087    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011100    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a7d957-d295-4ee4-9588-57819e924eeb-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.011289 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.011111    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da9c767d-9290-4cde-87d3-0d3bb6ecef8c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:22.409875 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.409840    7037 generic.go:356] "Generic (PLEG): container finished" podID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerID="a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a" exitCode=0
Oct 30 21:57:22.410087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.409885    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"
Oct 30 21:57:22.410087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.409925    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" event={"ID":"a0a7d957-d295-4ee4-9588-57819e924eeb","Type":"ContainerDied","Data":"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"}
Oct 30 21:57:22.410087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.409967    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk" event={"ID":"a0a7d957-d295-4ee4-9588-57819e924eeb","Type":"ContainerDied","Data":"66bab6ece5151b14e50fc275f24cf31c4041e2d7e60a10056a3b44f3acf957b8"}
Oct 30 21:57:22.410087 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.409988    7037 scope.go:122] "RemoveContainer" containerID="a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"
Oct 30 21:57:22.411387 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.411349    7037 generic.go:356] "Generic (PLEG): container finished" podID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerID="50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6" exitCode=0
Oct 30 21:57:22.411535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.411386    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"
Oct 30 21:57:22.411535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.411382    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" event={"ID":"da9c767d-9290-4cde-87d3-0d3bb6ecef8c","Type":"ContainerDied","Data":"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"}
Oct 30 21:57:22.411535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.411503    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" event={"ID":"da9c767d-9290-4cde-87d3-0d3bb6ecef8c","Type":"ContainerDied","Data":"0a3b40c335a735910f3114f17e2bbe4d5d4341f24462692f8a67ea1511b8f8f0"}
Oct 30 21:57:22.420222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.420205    7037 scope.go:122] "RemoveContainer" containerID="a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"
Oct 30 21:57:22.420531 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:22.420484    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a\": container with ID starting with a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a not found: ID does not exist" containerID="a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"
Oct 30 21:57:22.420531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.420526    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a"} err="failed to get container status \"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a\": rpc error: code = NotFound desc = could not find container \"a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a\": container with ID starting with a455e935c8320b6dd1d1b4f5cd3dd930adb465e356b5fe16e83ed3d59d2f854a not found: ID does not exist"
Oct 30 21:57:22.420712 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.420544    7037 scope.go:122] "RemoveContainer" containerID="50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"
Oct 30 21:57:22.430143 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.430126    7037 scope.go:122] "RemoveContainer" containerID="50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"
Oct 30 21:57:22.430399 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:22.430380    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6\": container with ID starting with 50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6 not found: ID does not exist" containerID="50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"
Oct 30 21:57:22.430447 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.430406    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6"} err="failed to get container status \"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6\": rpc error: code = NotFound desc = could not find container \"50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6\": container with ID starting with 50e89bd1d88247807b1051fc574696a99c3d98e242d0d93fad5adb7b64d4c2f6 not found: ID does not exist"
Oct 30 21:57:22.472353 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.472320    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:57:22.484842 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.484813    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt"]
Oct 30 21:57:22.511706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.511663    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:57:22.515704 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.515679    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-839163/distinct-w4-839163-kserve-6c4bbcd677-549nk"]
Oct 30 21:57:22.715561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.715469    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" path="/var/lib/kubelet/pods/a0a7d957-d295-4ee4-9588-57819e924eeb/volumes"
Oct 30 21:57:22.716024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.716001    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" path="/var/lib/kubelet/pods/da9c767d-9290-4cde-87d3-0d3bb6ecef8c/volumes"
Oct 30 21:57:22.813804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:22.813755    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/premium-w4-839163-kserve-8784b6c46-bf7pt" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.434802 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.434761    7037 generic.go:356] "Generic (PLEG): container finished" podID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerID="daafcf3b5d02b5a63ffaae8168e14fe3ebf4fbfc67b5e8cf080fc088deaec309" exitCode=0
Oct 30 21:57:25.435195 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.434835    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" event={"ID":"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1","Type":"ContainerDied","Data":"daafcf3b5d02b5a63ffaae8168e14fe3ebf4fbfc67b5e8cf080fc088deaec309"}
Oct 30 21:57:25.566134 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.566073    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:57:25.648348 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.648309    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.648555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.648356    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.648555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.648395    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.648555 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.648455    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.648989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.648698    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:25.649089 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649042    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.649147 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649093    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.649279 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649245    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.649365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649275    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:25.649365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649298    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.649365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649340    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\") pod \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\" (UID: \"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1\") "
Oct 30 21:57:25.649580 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649378    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:25.649898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649634    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:25.649898 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.649858    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:25.650174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.650145    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.650304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.650178    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.650304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.650193    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.650304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.650207    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.651257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.651230    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:25.651351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.651303    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:25.651535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.651518    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:25.651999 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.651984    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz" pod "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" (UID: "fb569ebc-e1ee-4c61-8fab-40f63e6c6be1"). InnerVolumeSpecName "kube-api-access-z47lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:25.751411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.751370    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.751411 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.751409    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.751664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.751425    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47lz\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-kube-api-access-z47lz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.751664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.751441    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:25.751664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:25.751453    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:26.581511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:26.574420    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82" event={"ID":"fb569ebc-e1ee-4c61-8fab-40f63e6c6be1","Type":"ContainerDied","Data":"9448c6e1203af463766053c1d443c24e8bb848df7d934a5bca5993d8aaa4a67a"}
Oct 30 21:57:26.581511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:26.574513    7037 scope.go:122] "RemoveContainer" containerID="daafcf3b5d02b5a63ffaae8168e14fe3ebf4fbfc67b5e8cf080fc088deaec309"
Oct 30 21:57:26.581511 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:26.574693    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"
Oct 30 21:57:26.765249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:26.765181    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:57:26.776984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:26.776947    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-c8w82"]
Oct 30 21:57:28.707242 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:28.707206    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" path="/var/lib/kubelet/pods/fb569ebc-e1ee-4c61-8fab-40f63e6c6be1/volumes"
Oct 30 21:57:30.344563 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:30.344528    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:30.344966 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:30.344571    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:33.672785 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:33.672740    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:57:37.428989 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:37.428949    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:57:37.429421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:37.429153    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" containerID="cri-o://d339c5aa135d7814fa1a8c59ba6d06f4839920dfbe031b52be0fd8157b28b65d" gracePeriod=30
Oct 30 21:57:38.937476 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:38.937427    7037 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:57:38.937920 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:38.937520    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:57:40.345044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:40.345004    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:40.345044 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:40.345026    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:41.087304 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.087269    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:57:41.123796 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.123767    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:57:41.194445 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194410    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\") pod \"68cca4ca-b814-4069-baf8-0f90f7768242\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") "
Oct 30 21:57:41.194655 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194554    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\") pod \"8ebb53be-383f-4ae5-ba21-322a2c35667f\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") "
Oct 30 21:57:41.194730 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194701    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\" (UniqueName: \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\") pod \"68cca4ca-b814-4069-baf8-0f90f7768242\" (UID: \"68cca4ca-b814-4069-baf8-0f90f7768242\") "
Oct 30 21:57:41.194786 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194734    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\" (UniqueName: \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\") pod \"8ebb53be-383f-4ae5-ba21-322a2c35667f\" (UID: \"8ebb53be-383f-4ae5-ba21-322a2c35667f\") "
Oct 30 21:57:41.194968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194925    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume" pod "68cca4ca-b814-4069-baf8-0f90f7768242" (UID: "68cca4ca-b814-4069-baf8-0f90f7768242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:41.194968 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.194947    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume" pod "8ebb53be-383f-4ae5-ba21-322a2c35667f" (UID: "8ebb53be-383f-4ae5-ba21-322a2c35667f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:41.195158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.195129    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cca4ca-b814-4069-baf8-0f90f7768242-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:41.195158 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.195150    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ebb53be-383f-4ae5-ba21-322a2c35667f-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:41.196861 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.196831    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x" pod "68cca4ca-b814-4069-baf8-0f90f7768242" (UID: "68cca4ca-b814-4069-baf8-0f90f7768242"). InnerVolumeSpecName "kube-api-access-zqf6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:41.196972 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.196885    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz" pod "8ebb53be-383f-4ae5-ba21-322a2c35667f" (UID: "8ebb53be-383f-4ae5-ba21-322a2c35667f"). InnerVolumeSpecName "kube-api-access-5lvgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:41.296403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.296366    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf6x\" (UniqueName: \"kubernetes.io/projected/68cca4ca-b814-4069-baf8-0f90f7768242-kube-api-access-zqf6x\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:41.296403 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.296401    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvgz\" (UniqueName: \"kubernetes.io/projected/8ebb53be-383f-4ae5-ba21-322a2c35667f-kube-api-access-5lvgz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:41.661663 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.661627    7037 generic.go:356] "Generic (PLEG): container finished" podID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerID="0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9" exitCode=0
Oct 30 21:57:41.662113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.661675    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"
Oct 30 21:57:41.662113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.661718    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" event={"ID":"8ebb53be-383f-4ae5-ba21-322a2c35667f","Type":"ContainerDied","Data":"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"}
Oct 30 21:57:41.662113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.661781    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q" event={"ID":"8ebb53be-383f-4ae5-ba21-322a2c35667f","Type":"ContainerDied","Data":"909386e3ea026d5427c763cd564023717bf098a4c0fd943d11c4972133f617f1"}
Oct 30 21:57:41.662113 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.661811    7037 scope.go:122] "RemoveContainer" containerID="0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"
Oct 30 21:57:41.663194 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.663172    7037 generic.go:356] "Generic (PLEG): container finished" podID="68cca4ca-b814-4069-baf8-0f90f7768242" containerID="3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c" exitCode=0
Oct 30 21:57:41.663303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.663222    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" event={"ID":"68cca4ca-b814-4069-baf8-0f90f7768242","Type":"ContainerDied","Data":"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"}
Oct 30 21:57:41.663303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.663272    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h" event={"ID":"68cca4ca-b814-4069-baf8-0f90f7768242","Type":"ContainerDied","Data":"67117eaea534e4a26b7b1534d5296fff3f05ce6073965e7cfe3c52f28fb8cfa3"}
Oct 30 21:57:41.663303 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.663231    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"
Oct 30 21:57:41.672410 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.672392    7037 scope.go:122] "RemoveContainer" containerID="0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"
Oct 30 21:57:41.672738 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:41.672717    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9\": container with ID starting with 0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9 not found: ID does not exist" containerID="0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"
Oct 30 21:57:41.672804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.672745    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9"} err="failed to get container status \"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9\": rpc error: code = NotFound desc = could not find container \"0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9\": container with ID starting with 0b4522f42ca7bd069e66a474f3ff8506e735a7690f3b88269f766816b9ee78f9 not found: ID does not exist"
Oct 30 21:57:41.672804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.672762    7037 scope.go:122] "RemoveContainer" containerID="3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"
Oct 30 21:57:41.681466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.681450    7037 scope.go:122] "RemoveContainer" containerID="3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"
Oct 30 21:57:41.681776 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:41.681757    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c\": container with ID starting with 3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c not found: ID does not exist" containerID="3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"
Oct 30 21:57:41.681855 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.681784    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c"} err="failed to get container status \"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c\": rpc error: code = NotFound desc = could not find container \"3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c\": container with ID starting with 3e8024ccbf65b6edd087ef16973ca1c84744e3e1a2ee61015f1882504f06c99c not found: ID does not exist"
Oct 30 21:57:41.721616 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.721581    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:57:41.728433 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.728403    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-da52fd-7b88fccc79-btv5h"]
Oct 30 21:57:41.776584 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.776553    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:57:41.789598 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.789568    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-da52fd-8567bc8548-6nh6q"]
Oct 30 21:57:41.891455 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:41.891422    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:57:42.670722 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.670693    7037 generic.go:356] "Generic (PLEG): container finished" podID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerID="d339c5aa135d7814fa1a8c59ba6d06f4839920dfbe031b52be0fd8157b28b65d" exitCode=0
Oct 30 21:57:42.671132 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.670781    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" event={"ID":"d3aeb557-2e9d-404a-bfb1-be2b127efc10","Type":"ContainerDied","Data":"d339c5aa135d7814fa1a8c59ba6d06f4839920dfbe031b52be0fd8157b28b65d"}
Oct 30 21:57:42.695201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.695177    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:57:42.706662 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.706586    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" path="/var/lib/kubelet/pods/68cca4ca-b814-4069-baf8-0f90f7768242/volumes"
Oct 30 21:57:42.707025 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.707008    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" path="/var/lib/kubelet/pods/8ebb53be-383f-4ae5-ba21-322a2c35667f/volumes"
Oct 30 21:57:42.812998 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.812962    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813028    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813075    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813110    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813141    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813162    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813199 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813191    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813225    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813259    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\") pod \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\" (UID: \"d3aeb557-2e9d-404a-bfb1-be2b127efc10\") "
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813310    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813324    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813423    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:42.813599 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813569    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:42.813916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813700    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:42.814001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.813979    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.814066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.814007    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.814066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.814021    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.814066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.814035    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.814066 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.814050    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.815890 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.815849    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "kube-api-access-9jttx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:42.816041 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.816021    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:42.816128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.816060    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:42.816128 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.816094    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo" pod "d3aeb557-2e9d-404a-bfb1-be2b127efc10" (UID: "d3aeb557-2e9d-404a-bfb1-be2b127efc10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:42.915617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.915583    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.915617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.915613    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.915617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.915623    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jttx\" (UniqueName: \"kubernetes.io/projected/d3aeb557-2e9d-404a-bfb1-be2b127efc10-kube-api-access-9jttx\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.915617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:42.915632    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3aeb557-2e9d-404a-bfb1-be2b127efc10-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:43.680570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:43.680538    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"
Oct 30 21:57:43.680570 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:43.680547    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg" event={"ID":"d3aeb557-2e9d-404a-bfb1-be2b127efc10","Type":"ContainerDied","Data":"1d4f5cd45361b43abd6c6ae18c2779d66309d35bdc840d1f6b3d416c9f7f71e0"}
Oct 30 21:57:43.681021 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:43.680618    7037 scope.go:122] "RemoveContainer" containerID="d339c5aa135d7814fa1a8c59ba6d06f4839920dfbe031b52be0fd8157b28b65d"
Oct 30 21:57:43.750857 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:43.750821    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:57:43.763367 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:43.763329    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-rjmgg"]
Oct 30 21:57:44.710702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:44.707768    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" path="/var/lib/kubelet/pods/d3aeb557-2e9d-404a-bfb1-be2b127efc10/volumes"
Oct 30 21:57:47.646280 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:47.646240    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:57:47.684539 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:47.684500    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:57:52.479017 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:52.478977    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:57:52.545160 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:52.545124    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:57:52.647449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:52.647407    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing" containerID="cri-o://59609f3f5bec595acbf7a7b1550ae4febcba6725f51d77e31dbab15861b83381" gracePeriod=25
Oct 30 21:57:52.684849 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:52.684792    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing" containerID="cri-o://f538b97c8f33c33752478f9055243327ccc1829b2536cf409105fd1c73f2300b" gracePeriod=25
Oct 30 21:57:56.912822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:56.912781    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main" containerID="cri-o://fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5" gracePeriod=45
Oct 30 21:57:57.173416 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.173358    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:57:57.277407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277376    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277407 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277420    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277467    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\" (UniqueName: \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277519    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277625    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277695 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277659    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\" (UniqueName: \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\") pod \"7700176f-5342-4db9-bc74-f86befc42f39\" (UID: \"7700176f-5342-4db9-bc74-f86befc42f39\") "
Oct 30 21:57:57.277923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277728    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:57.277923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.277786    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:57.278156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.278131    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.278156 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.278157    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.278420 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.278391    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:57.281535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.281429    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:57.281535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.281439    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:57.281535 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.281485    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8" pod "7700176f-5342-4db9-bc74-f86befc42f39" (UID: "7700176f-5342-4db9-bc74-f86befc42f39"). InnerVolumeSpecName "kube-api-access-f2bd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:57.321254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.321215    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:57.321485 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.321222    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:57.378949 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.378917    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2bd8\" (UniqueName: \"kubernetes.io/projected/7700176f-5342-4db9-bc74-f86befc42f39-kube-api-access-f2bd8\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.379133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.378970    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.379133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.378984    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7700176f-5342-4db9-bc74-f86befc42f39-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.379133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.378996    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7700176f-5342-4db9-bc74-f86befc42f39-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.480192 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.480093    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing" containerID="cri-o://4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3" gracePeriod=25
Oct 30 21:57:57.545919 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.545879    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing" containerID="cri-o://10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837" gracePeriod=25
Oct 30 21:57:57.765881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.765794    7037 generic.go:356] "Generic (PLEG): container finished" podID="7700176f-5342-4db9-bc74-f86befc42f39" containerID="fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5" exitCode=0
Oct 30 21:57:57.765881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.765840    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"
Oct 30 21:57:57.765881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.765849    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" event={"ID":"7700176f-5342-4db9-bc74-f86befc42f39","Type":"ContainerDied","Data":"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"}
Oct 30 21:57:57.765881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.765880    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt" event={"ID":"7700176f-5342-4db9-bc74-f86befc42f39","Type":"ContainerDied","Data":"5a455d1e1380a201927ff8e66266252aee43b4775bfe78e6c67c31d79930d4f5"}
Oct 30 21:57:57.766192 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.765896    7037 scope.go:122] "RemoveContainer" containerID="fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"
Oct 30 21:57:57.788560 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.788527    7037 scope.go:122] "RemoveContainer" containerID="fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"
Oct 30 21:57:57.791483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.791455    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:57:57.791714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.791690    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" podUID="708f1d12-9691-4eed-ac7d-2b34dab74c27" containerName="maas-api" containerID="cri-o://467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a" gracePeriod=30
Oct 30 21:57:57.796642 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:57.795758    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5\": container with ID starting with fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5 not found: ID does not exist" containerID="fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"
Oct 30 21:57:57.796642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.795843    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5"} err="failed to get container status \"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5\": rpc error: code = NotFound desc = could not find container \"fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5\": container with ID starting with fefc0fb12cd1bf9f649864581d012b45f7ddbf921819eafafa899b8d1e2090e5 not found: ID does not exist"
Oct 30 21:57:57.895275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.895242    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:57:57.902191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:57.902162    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/premium-w5-43c070-kserve-584cdcc7f8-dhcrt"]
Oct 30 21:57:58.044813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.044779    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:57:58.188513 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.188455    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\") pod \"708f1d12-9691-4eed-ac7d-2b34dab74c27\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") "
Oct 30 21:57:58.188790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.188546    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\" (UniqueName: \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\") pod \"708f1d12-9691-4eed-ac7d-2b34dab74c27\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") "
Oct 30 21:57:58.188790 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.188748    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\") pod \"708f1d12-9691-4eed-ac7d-2b34dab74c27\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") "
Oct 30 21:57:58.188928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.188796    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\") pod \"708f1d12-9691-4eed-ac7d-2b34dab74c27\" (UID: \"708f1d12-9691-4eed-ac7d-2b34dab74c27\") "
Oct 30 21:57:58.188928 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.188883    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca" pod "708f1d12-9691-4eed-ac7d-2b34dab74c27" (UID: "708f1d12-9691-4eed-ac7d-2b34dab74c27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:58.189159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.189139    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f1d12-9691-4eed-ac7d-2b34dab74c27-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.190734 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.190703    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj" pod "708f1d12-9691-4eed-ac7d-2b34dab74c27" (UID: "708f1d12-9691-4eed-ac7d-2b34dab74c27"). InnerVolumeSpecName "kube-api-access-nd7wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:58.190848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.190832    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls" pod "708f1d12-9691-4eed-ac7d-2b34dab74c27" (UID: "708f1d12-9691-4eed-ac7d-2b34dab74c27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:58.190896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.190853    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs" pod "708f1d12-9691-4eed-ac7d-2b34dab74c27" (UID: "708f1d12-9691-4eed-ac7d-2b34dab74c27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:58.290471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.290390    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7wj\" (UniqueName: \"kubernetes.io/projected/708f1d12-9691-4eed-ac7d-2b34dab74c27-kube-api-access-nd7wj\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.290471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.290423    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.290471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.290433    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f1d12-9691-4eed-ac7d-2b34dab74c27-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.706886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.706848    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7700176f-5342-4db9-bc74-f86befc42f39" path="/var/lib/kubelet/pods/7700176f-5342-4db9-bc74-f86befc42f39/volumes"
Oct 30 21:57:58.773177 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.773146    7037 generic.go:356] "Generic (PLEG): container finished" podID="708f1d12-9691-4eed-ac7d-2b34dab74c27" containerID="467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a" exitCode=0
Oct 30 21:57:58.773390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.773195    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"
Oct 30 21:57:58.773390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.773230    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" event={"ID":"708f1d12-9691-4eed-ac7d-2b34dab74c27","Type":"ContainerDied","Data":"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"}
Oct 30 21:57:58.773390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.773283    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd" event={"ID":"708f1d12-9691-4eed-ac7d-2b34dab74c27","Type":"ContainerDied","Data":"ecc6b8c788873e65ecf45b364c66e58b0ab1e82769cf69da357541254a073f76"}
Oct 30 21:57:58.773390 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.773308    7037 scope.go:122] "RemoveContainer" containerID="467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"
Oct 30 21:57:58.783161 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.783143    7037 scope.go:122] "RemoveContainer" containerID="467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"
Oct 30 21:57:58.783513 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:57:58.783462    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a\": container with ID starting with 467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a not found: ID does not exist" containerID="467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"
Oct 30 21:57:58.783626 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.783519    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a"} err="failed to get container status \"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a\": rpc error: code = NotFound desc = could not find container \"467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a\": container with ID starting with 467893843f66a8a740c028e609c6ff340025f94e9f8868394b038857041e204a not found: ID does not exist"
Oct 30 21:57:58.824558 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.824525    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:57:58.836952 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:57:58.836923    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-908469-759db5b94-l8dtd"]
Oct 30 21:58:00.707249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:00.707212    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708f1d12-9691-4eed-ac7d-2b34dab74c27" path="/var/lib/kubelet/pods/708f1d12-9691-4eed-ac7d-2b34dab74c27/volumes"
Oct 30 21:58:03.671834 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:03.671789    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:58:06.108056 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:06.107970    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:06.108483 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:06.107965    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:07.321768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:07.321721    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:07.322361 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:07.322336    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:10.905251 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:10.905216    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:58:10.905744 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:10.905477    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" podUID="8e36be97-86a1-49d3-b9b9-290c7395730d" containerName="maas-api" containerID="cri-o://479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838" gracePeriod=30
Oct 30 21:58:11.159610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.159543    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:58:11.215084 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215049    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") pod \"8e36be97-86a1-49d3-b9b9-290c7395730d\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") "
Oct 30 21:58:11.215258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215135    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\" (UniqueName: \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\") pod \"8e36be97-86a1-49d3-b9b9-290c7395730d\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") "
Oct 30 21:58:11.215258 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215249    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\") pod \"8e36be97-86a1-49d3-b9b9-290c7395730d\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") "
Oct 30 21:58:11.215348 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215300    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\") pod \"8e36be97-86a1-49d3-b9b9-290c7395730d\" (UID: \"8e36be97-86a1-49d3-b9b9-290c7395730d\") "
Oct 30 21:58:11.215643 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215610    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca" pod "8e36be97-86a1-49d3-b9b9-290c7395730d" (UID: "8e36be97-86a1-49d3-b9b9-290c7395730d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:11.215804 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.215736    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e36be97-86a1-49d3-b9b9-290c7395730d-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:11.217260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.217230    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls" pod "8e36be97-86a1-49d3-b9b9-290c7395730d" (UID: "8e36be97-86a1-49d3-b9b9-290c7395730d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:11.217260 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.217250    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs" pod "8e36be97-86a1-49d3-b9b9-290c7395730d" (UID: "8e36be97-86a1-49d3-b9b9-290c7395730d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:11.217439 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.217365    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9" pod "8e36be97-86a1-49d3-b9b9-290c7395730d" (UID: "8e36be97-86a1-49d3-b9b9-290c7395730d"). InnerVolumeSpecName "kube-api-access-srdw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:11.316787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.316750    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdw9\" (UniqueName: \"kubernetes.io/projected/8e36be97-86a1-49d3-b9b9-290c7395730d-kube-api-access-srdw9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:11.316787 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.316781    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:11.316979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.316796    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e36be97-86a1-49d3-b9b9-290c7395730d-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:11.851664 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.851626    7037 generic.go:356] "Generic (PLEG): container finished" podID="8e36be97-86a1-49d3-b9b9-290c7395730d" containerID="479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838" exitCode=0
Oct 30 21:58:11.851864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.851669    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"
Oct 30 21:58:11.851864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.851686    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" event={"ID":"8e36be97-86a1-49d3-b9b9-290c7395730d","Type":"ContainerDied","Data":"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"}
Oct 30 21:58:11.851864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.851735    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5" event={"ID":"8e36be97-86a1-49d3-b9b9-290c7395730d","Type":"ContainerDied","Data":"9f54d7814990df65b17dd9c887b691d67411ceb3073f24b0a85ae78e9547b096"}
Oct 30 21:58:11.851864 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.851756    7037 scope.go:122] "RemoveContainer" containerID="479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"
Oct 30 21:58:11.862002 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.861979    7037 scope.go:122] "RemoveContainer" containerID="479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"
Oct 30 21:58:11.862303 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:11.862282    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838\": container with ID starting with 479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838 not found: ID does not exist" containerID="479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"
Oct 30 21:58:11.862351 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.862312    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838"} err="failed to get container status \"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838\": rpc error: code = NotFound desc = could not find container \"479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838\": container with ID starting with 479c5ea330d67657efeb6a78ec6b25324f087557fabfe020218484b44e487838 not found: ID does not exist"
Oct 30 21:58:11.912911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.912883    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:58:11.925399 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:11.925369    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-43c070-68db4db79c-ggqr5"]
Oct 30 21:58:12.706899 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.706867    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e36be97-86a1-49d3-b9b9-290c7395730d" path="/var/lib/kubelet/pods/8e36be97-86a1-49d3-b9b9-290c7395730d/volumes"
Oct 30 21:58:12.867762 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.867729    7037 generic.go:356] "Generic (PLEG): container finished" podID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerID="f538b97c8f33c33752478f9055243327ccc1829b2536cf409105fd1c73f2300b" exitCode=0
Oct 30 21:58:12.867959 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.867804    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" event={"ID":"f5a1927f-7fa5-4080-9386-e0227138f6bc","Type":"ContainerDied","Data":"f538b97c8f33c33752478f9055243327ccc1829b2536cf409105fd1c73f2300b"}
Oct 30 21:58:12.869224 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.869181    7037 generic.go:356] "Generic (PLEG): container finished" podID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerID="59609f3f5bec595acbf7a7b1550ae4febcba6725f51d77e31dbab15861b83381" exitCode=0
Oct 30 21:58:12.869370 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.869231    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" event={"ID":"11a4337b-76c0-4bbb-af4b-4098ac252864","Type":"ContainerDied","Data":"59609f3f5bec595acbf7a7b1550ae4febcba6725f51d77e31dbab15861b83381"}
Oct 30 21:58:12.945207 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.945180    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:58:12.975057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:12.975029    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:58:13.037169 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037137    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\") pod \"f5a1927f-7fa5-4080-9386-e0227138f6bc\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") "
Oct 30 21:58:13.037362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037239    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\" (UniqueName: \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\") pod \"f5a1927f-7fa5-4080-9386-e0227138f6bc\" (UID: \"f5a1927f-7fa5-4080-9386-e0227138f6bc\") "
Oct 30 21:58:13.037362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037267    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\") pod \"11a4337b-76c0-4bbb-af4b-4098ac252864\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") "
Oct 30 21:58:13.037362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037342    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\" (UniqueName: \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\") pod \"11a4337b-76c0-4bbb-af4b-4098ac252864\" (UID: \"11a4337b-76c0-4bbb-af4b-4098ac252864\") "
Oct 30 21:58:13.037671 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037642    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume" pod "f5a1927f-7fa5-4080-9386-e0227138f6bc" (UID: "f5a1927f-7fa5-4080-9386-e0227138f6bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:13.037753 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.037731    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume" pod "11a4337b-76c0-4bbb-af4b-4098ac252864" (UID: "11a4337b-76c0-4bbb-af4b-4098ac252864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:13.039923 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.039898    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7" pod "11a4337b-76c0-4bbb-af4b-4098ac252864" (UID: "11a4337b-76c0-4bbb-af4b-4098ac252864"). InnerVolumeSpecName "kube-api-access-wkzq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:13.040050 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.039960    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k" pod "f5a1927f-7fa5-4080-9386-e0227138f6bc" (UID: "f5a1927f-7fa5-4080-9386-e0227138f6bc"). InnerVolumeSpecName "kube-api-access-t5v8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:13.138499 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.138465    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a1927f-7fa5-4080-9386-e0227138f6bc-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:13.138672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.138528    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5v8k\" (UniqueName: \"kubernetes.io/projected/f5a1927f-7fa5-4080-9386-e0227138f6bc-kube-api-access-t5v8k\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:13.138672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.138545    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a4337b-76c0-4bbb-af4b-4098ac252864-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:13.138672 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.138558    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzq7\" (UniqueName: \"kubernetes.io/projected/11a4337b-76c0-4bbb-af4b-4098ac252864-kube-api-access-wkzq7\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:13.876061 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.876032    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"
Oct 30 21:58:13.876263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.876065    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96" event={"ID":"11a4337b-76c0-4bbb-af4b-4098ac252864","Type":"ContainerDied","Data":"46f15bab3d6418dd58fc8910d668ec863684d8d39929672f774fb9b097b54496"}
Oct 30 21:58:13.876263 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.876125    7037 scope.go:122] "RemoveContainer" containerID="59609f3f5bec595acbf7a7b1550ae4febcba6725f51d77e31dbab15861b83381"
Oct 30 21:58:13.877715 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.877691    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j" event={"ID":"f5a1927f-7fa5-4080-9386-e0227138f6bc","Type":"ContainerDied","Data":"5ecfaea8895cb029cfd5c5fd65147f3aade5d7b7bd77c3196731e683d87a1a07"}
Oct 30 21:58:13.877808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.877718    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"
Oct 30 21:58:13.885916 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.885897    7037 scope.go:122] "RemoveContainer" containerID="f538b97c8f33c33752478f9055243327ccc1829b2536cf409105fd1c73f2300b"
Oct 30 21:58:13.935103 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.935016    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:58:13.947401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.947369    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-43c070-78859bb9c8-cnl96"]
Oct 30 21:58:13.980625 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.980589    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:58:13.987145 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:13.987119    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-43c070-6f8cb67579-pp85j"]
Oct 30 21:58:14.705872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:14.705837    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" path="/var/lib/kubelet/pods/11a4337b-76c0-4bbb-af4b-4098ac252864/volumes"
Oct 30 21:58:14.706228 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:14.706215    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" path="/var/lib/kubelet/pods/f5a1927f-7fa5-4080-9386-e0227138f6bc/volumes"
Oct 30 21:58:16.107438 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:16.107399    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:16.107950 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:16.107399    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:17.102179 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.102140    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:58:17.144714 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.144678    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:58:17.306000 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.305965    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:58:17.306223 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.306191    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" podUID="ce9b6918-d7ce-4a16-b058-298e2f660177" containerName="maas-api" containerID="cri-o://d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1" gracePeriod=30
Oct 30 21:58:17.671001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.670716    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:58:17.787429 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.787403    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\" (UniqueName: \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\") pod \"ce9b6918-d7ce-4a16-b058-298e2f660177\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") "
Oct 30 21:58:17.787565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.787464    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\") pod \"ce9b6918-d7ce-4a16-b058-298e2f660177\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") "
Oct 30 21:58:17.787565 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.787529    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\") pod \"ce9b6918-d7ce-4a16-b058-298e2f660177\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") "
Oct 30 21:58:17.787647 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.787618    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\") pod \"ce9b6918-d7ce-4a16-b058-298e2f660177\" (UID: \"ce9b6918-d7ce-4a16-b058-298e2f660177\") "
Oct 30 21:58:17.788133 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.788105    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca" pod "ce9b6918-d7ce-4a16-b058-298e2f660177" (UID: "ce9b6918-d7ce-4a16-b058-298e2f660177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:17.790419 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.790395    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm" pod "ce9b6918-d7ce-4a16-b058-298e2f660177" (UID: "ce9b6918-d7ce-4a16-b058-298e2f660177"). InnerVolumeSpecName "kube-api-access-lkxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:17.790891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.790856    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls" pod "ce9b6918-d7ce-4a16-b058-298e2f660177" (UID: "ce9b6918-d7ce-4a16-b058-298e2f660177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:17.796094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.796063    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs" pod "ce9b6918-d7ce-4a16-b058-298e2f660177" (UID: "ce9b6918-d7ce-4a16-b058-298e2f660177"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:17.796706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.796681    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:58:17.843910 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.843880    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:58:17.888418 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.888384    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\") pod \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") "
Oct 30 21:58:17.888605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.888550    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\" (UniqueName: \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\") pod \"248fa86b-6260-4f2a-87b8-12110595f28b\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") "
Oct 30 21:58:17.888605 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.888585    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\") pod \"248fa86b-6260-4f2a-87b8-12110595f28b\" (UID: \"248fa86b-6260-4f2a-87b8-12110595f28b\") "
Oct 30 21:58:17.888733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.888629    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\" (UniqueName: \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\") pod \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\" (UID: \"785797e8-bb2d-4ed3-86ed-68a801f1a38f\") "
Oct 30 21:58:17.888834 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.888811    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume" pod "785797e8-bb2d-4ed3-86ed-68a801f1a38f" (UID: "785797e8-bb2d-4ed3-86ed-68a801f1a38f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:17.889275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889235    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785797e8-bb2d-4ed3-86ed-68a801f1a38f-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.889275 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889267    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxwm\" (UniqueName: \"kubernetes.io/projected/ce9b6918-d7ce-4a16-b058-298e2f660177-kube-api-access-lkxwm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.889470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889283    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9b6918-d7ce-4a16-b058-298e2f660177-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.889470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889300    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.889470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889296    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume" pod "248fa86b-6260-4f2a-87b8-12110595f28b" (UID: "248fa86b-6260-4f2a-87b8-12110595f28b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:17.889470 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.889313    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9b6918-d7ce-4a16-b058-298e2f660177-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.890967 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.890945    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j" pod "248fa86b-6260-4f2a-87b8-12110595f28b" (UID: "248fa86b-6260-4f2a-87b8-12110595f28b"). InnerVolumeSpecName "kube-api-access-82p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:17.891073 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.891036    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn" pod "785797e8-bb2d-4ed3-86ed-68a801f1a38f" (UID: "785797e8-bb2d-4ed3-86ed-68a801f1a38f"). InnerVolumeSpecName "kube-api-access-7cprn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:17.901881 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.901854    7037 generic.go:356] "Generic (PLEG): container finished" podID="248fa86b-6260-4f2a-87b8-12110595f28b" containerID="10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837" exitCode=0
Oct 30 21:58:17.901992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.901901    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"
Oct 30 21:58:17.901992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.901923    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" event={"ID":"248fa86b-6260-4f2a-87b8-12110595f28b","Type":"ContainerDied","Data":"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"}
Oct 30 21:58:17.901992 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.901973    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2" event={"ID":"248fa86b-6260-4f2a-87b8-12110595f28b","Type":"ContainerDied","Data":"664076d40252cca311e7e7928c0abc9bb77ecff4dcf9230b13fd64f9d0760ba0"}
Oct 30 21:58:17.902153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.902005    7037 scope.go:122] "RemoveContainer" containerID="10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"
Oct 30 21:58:17.903516 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.903461    7037 generic.go:356] "Generic (PLEG): container finished" podID="ce9b6918-d7ce-4a16-b058-298e2f660177" containerID="d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1" exitCode=0
Oct 30 21:58:17.903617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.903533    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"
Oct 30 21:58:17.903617 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.903591    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" event={"ID":"ce9b6918-d7ce-4a16-b058-298e2f660177","Type":"ContainerDied","Data":"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"}
Oct 30 21:58:17.903745 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.903644    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb" event={"ID":"ce9b6918-d7ce-4a16-b058-298e2f660177","Type":"ContainerDied","Data":"5fd5086f902b0beb1c183fb589bcde06c1370f319049e1d71487f7dc66dc8605"}
Oct 30 21:58:17.905190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.905165    7037 generic.go:356] "Generic (PLEG): container finished" podID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerID="4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3" exitCode=0
Oct 30 21:58:17.905311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.905209    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"
Oct 30 21:58:17.905311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.905233    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" event={"ID":"785797e8-bb2d-4ed3-86ed-68a801f1a38f","Type":"ContainerDied","Data":"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"}
Oct 30 21:58:17.905311 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.905276    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7" event={"ID":"785797e8-bb2d-4ed3-86ed-68a801f1a38f","Type":"ContainerDied","Data":"a98d0d2d22960d1b64bf87a893ae822a3be8372b1dee298a7a8f10433d085351"}
Oct 30 21:58:17.917110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.917090    7037 scope.go:122] "RemoveContainer" containerID="10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"
Oct 30 21:58:17.917431 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:17.917411    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837\": container with ID starting with 10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837 not found: ID does not exist" containerID="10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"
Oct 30 21:58:17.917537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.917438    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837"} err="failed to get container status \"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837\": rpc error: code = NotFound desc = could not find container \"10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837\": container with ID starting with 10999ffde9d491bfe861419908b5ead03c9982bf90d8a38261fe30c2847e2837 not found: ID does not exist"
Oct 30 21:58:17.917537 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.917467    7037 scope.go:122] "RemoveContainer" containerID="d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"
Oct 30 21:58:17.926706 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.926683    7037 scope.go:122] "RemoveContainer" containerID="d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"
Oct 30 21:58:17.926961 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:17.926941    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1\": container with ID starting with d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1 not found: ID does not exist" containerID="d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"
Oct 30 21:58:17.927028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.926967    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1"} err="failed to get container status \"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1\": rpc error: code = NotFound desc = could not find container \"d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1\": container with ID starting with d664a5b2d02b03bff570812de0a06273474c1c3834ca905af5d11f54ec88a5f1 not found: ID does not exist"
Oct 30 21:58:17.927028 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.926983    7037 scope.go:122] "RemoveContainer" containerID="4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"
Oct 30 21:58:17.936600 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.936582    7037 scope.go:122] "RemoveContainer" containerID="4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"
Oct 30 21:58:17.936870 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:17.936852    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3\": container with ID starting with 4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3 not found: ID does not exist" containerID="4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"
Oct 30 21:58:17.936912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.936877    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3"} err="failed to get container status \"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3\": rpc error: code = NotFound desc = could not find container \"4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3\": container with ID starting with 4eeb7ed02d77220c5a6dde76365326b836af7372a9e3b5f75a8c73d6525cb8c3 not found: ID does not exist"
Oct 30 21:58:17.982948 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.982872    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:58:17.988327 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.988299    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-908469-c575ddf9f-ll8n2"]
Oct 30 21:58:17.990294 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.990266    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82p5j\" (UniqueName: \"kubernetes.io/projected/248fa86b-6260-4f2a-87b8-12110595f28b-kube-api-access-82p5j\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.990395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.990295    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248fa86b-6260-4f2a-87b8-12110595f28b-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:17.990395 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:17.990307    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cprn\" (UniqueName: \"kubernetes.io/projected/785797e8-bb2d-4ed3-86ed-68a801f1a38f-kube-api-access-7cprn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:18.023702 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.023675    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:58:18.031031 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.028417    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-908469-994779fdd-55mn7"]
Oct 30 21:58:18.061053 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.061015    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:58:18.065909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.065882    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95692f-849f998576-9b5bb"]
Oct 30 21:58:18.706576 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.706504    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" path="/var/lib/kubelet/pods/248fa86b-6260-4f2a-87b8-12110595f28b/volumes"
Oct 30 21:58:18.706933 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.706891    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" path="/var/lib/kubelet/pods/785797e8-bb2d-4ed3-86ed-68a801f1a38f/volumes"
Oct 30 21:58:18.707247 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:18.707234    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9b6918-d7ce-4a16-b058-298e2f660177" path="/var/lib/kubelet/pods/ce9b6918-d7ce-4a16-b058-298e2f660177/volumes"
Oct 30 21:58:21.063378 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:21.063342    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:58:21.063813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:21.063667    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" containerID="cri-o://d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230" gracePeriod=30
Oct 30 21:58:22.103374 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:22.103314    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing" containerID="cri-o://c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9" gracePeriod=25
Oct 30 21:58:22.145903 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:22.145848    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing" containerID="cri-o://a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838" gracePeriod=25
Oct 30 21:58:24.104982 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:24.104944    7037 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:58:24.105531 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:24.105001    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:58:26.322675 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.322647    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:58:26.481909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.481823    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.481909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.481868    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.481909 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.481910    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.481962    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.481983    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482008    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482042    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482086    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482157    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\") pod \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\" (UID: \"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d\") "
Oct 30 21:58:26.482564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482363    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:26.482564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482376    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:26.482564 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482394    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:26.482733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482628    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.482733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482649    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.482733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482662    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.482733 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482668    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:26.482954 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.482740    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:26.484249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.484219    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:26.484362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.484265    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:26.484362 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.484284    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "kube-api-access-r9tpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:26.484449 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.484435    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo" pod "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" (UID: "ef0f73ea-b35a-4aa1-952e-c21b7d09c25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:58:26.528320 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528277    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"]
Oct 30 21:58:26.528822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528788    7037 cpu_manager.go:414] "containerMap: removing container" podUID="856ee13a-a04c-468f-a277-163bf956e0de" containerName="payload-pre-processing"
Oct 30 21:58:26.528822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528811    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856ee13a-a04c-468f-a277-163bf956e0de" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528833    7037 cpu_manager.go:414] "containerMap: removing container" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528841    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528854    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528862    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528873    7037 cpu_manager.go:414] "containerMap: removing container" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528880    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528901    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528909    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528918    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528924    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528936    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528944    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528968    7037 cpu_manager.go:414] "containerMap: removing container" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528977    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528987    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.528994    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy"
Oct 30 21:58:26.529001 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529006    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7766e-85f0-47ef-a75f-dd2e064114db" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529012    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7766e-85f0-47ef-a75f-dd2e064114db" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529027    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529035    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529045    7037 cpu_manager.go:414] "containerMap: removing container" podUID="9e24018a-efff-4258-aebd-b3722015e5d3" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529052    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e24018a-efff-4258-aebd-b3722015e5d3" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529065    7037 cpu_manager.go:414] "containerMap: removing container" podUID="967aa72f-f0ad-4f40-8559-37a251c093c3" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529072    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967aa72f-f0ad-4f40-8559-37a251c093c3" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529081    7037 cpu_manager.go:414] "containerMap: removing container" podUID="02d09b60-e573-40c6-9d22-bdd339728f14" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529088    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d09b60-e573-40c6-9d22-bdd339728f14" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529105    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529112    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529123    7037 cpu_manager.go:414] "containerMap: removing container" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529130    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529142    7037 cpu_manager.go:414] "containerMap: removing container" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529149    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529161    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529168    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529178    7037 cpu_manager.go:414] "containerMap: removing container" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529185    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529196    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529202    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529213    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8e36be97-86a1-49d3-b9b9-290c7395730d" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529219    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e36be97-86a1-49d3-b9b9-290c7395730d" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529231    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ce9b6918-d7ce-4a16-b058-298e2f660177" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529238    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9b6918-d7ce-4a16-b058-298e2f660177" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529253    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529261    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529273    7037 cpu_manager.go:414] "containerMap: removing container" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529281    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529293    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529301    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529316    7037 cpu_manager.go:414] "containerMap: removing container" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529324    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529336    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529343    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529354    7037 cpu_manager.go:414] "containerMap: removing container" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529361    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529373    7037 cpu_manager.go:414] "containerMap: removing container" podUID="708f1d12-9691-4eed-ac7d-2b34dab74c27" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529380    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f1d12-9691-4eed-ac7d-2b34dab74c27" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529392    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529400    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529410    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529416    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529429    7037 cpu_manager.go:414] "containerMap: removing container" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529437    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529447    7037 cpu_manager.go:414] "containerMap: removing container" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529455    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529466    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529474    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529503    7037 cpu_manager.go:414] "containerMap: removing container" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529510    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529520    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529527    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529536    7037 cpu_manager.go:414] "containerMap: removing container" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529544    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529569    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e7716237-5b0f-43aa-824f-1a74121d9a29" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529578    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7716237-5b0f-43aa-824f-1a74121d9a29" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529692    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7716237-5b0f-43aa-824f-1a74121d9a29" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529710    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="856ee13a-a04c-468f-a277-163bf956e0de" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529721    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="248fa86b-6260-4f2a-87b8-12110595f28b" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529733    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8b7bcb-cc21-4d8f-94ad-1991eb46ab83" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529744    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a6dca-9c9e-44eb-9b63-58bf541f80ba" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529755    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a1927f-7fa5-4080-9386-e0227138f6bc" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529764    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529771    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d67858-28fe-4705-9b95-567c6d9acc57" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529781    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ddf27d-ae2e-4c81-9153-060f55931652" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529790    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a0b1b2-b5bb-412c-87fc-5bc3492a709d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529799    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7766e-85f0-47ef-a75f-dd2e064114db" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529809    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75e3b69-f5d2-4bc9-95b8-ebf11602e7c0" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529819    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e24018a-efff-4258-aebd-b3722015e5d3" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529828    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="967aa72f-f0ad-4f40-8559-37a251c093c3" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529838    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d09b60-e573-40c6-9d22-bdd339728f14" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529848    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55e7476-3fb6-41e5-ad6e-d7b13bc414cc" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529857    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cca4ca-b814-4069-baf8-0f90f7768242" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529866    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aeb557-2e9d-404a-bfb1-be2b127efc10" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529877    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1d52d9-d0cd-4322-969b-3a3e841db30f" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529887    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9c767d-9290-4cde-87d3-0d3bb6ecef8c" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529896    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a7d957-d295-4ee4-9588-57819e924eeb" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529905    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e36be97-86a1-49d3-b9b9-290c7395730d" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529913    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9b6918-d7ce-4a16-b058-298e2f660177" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529924    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebb53be-383f-4ae5-ba21-322a2c35667f" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529933    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="df72b143-ed3f-4652-9c1d-cf400e8458a9" containerName="payload-pre-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529942    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59227f7-56e1-4111-982c-2d1c3713368b" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529950    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a4337b-76c0-4bbb-af4b-4098ac252864" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529959    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529967    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529975    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="696591a5-faa5-4462-9994-292c599c6030" containerName="payload-processing"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529983    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f1d12-9691-4eed-ac7d-2b34dab74c27" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529990    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad803c3d-e2e8-400f-995b-c81cd7d60772" containerName="main"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.529998    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e3def0-0bfc-4501-95c7-bafc91844aef" containerName="maas-api"
Oct 30 21:58:26.529955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530006    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d9b503-6346-439a-81be-4fb943579121" containerName="payload-processing"
Oct 30 21:58:26.533468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530015    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb569ebc-e1ee-4c61-8fab-40f63e6c6be1" containerName="istio-proxy"
Oct 30 21:58:26.533468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530025    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7700176f-5342-4db9-bc74-f86befc42f39" containerName="main"
Oct 30 21:58:26.533468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530034    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="318c1981-a3b1-4528-9229-c46921459fb4" containerName="payload-pre-processing"
Oct 30 21:58:26.533468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530044    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70fdb94-df8e-4e9c-aa5d-1c924113345d" containerName="istio-proxy"
Oct 30 21:58:26.533468 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.530052    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="785797e8-bb2d-4ed3-86ed-68a801f1a38f" containerName="payload-processing"
Oct 30 21:58:26.533979 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.533960    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.557638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.549997    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:26.557638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.550118    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:26.557638 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.551410    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"]
Oct 30 21:58:26.583554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583521    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.583554 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583553    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.583779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583567    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.583779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583579    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tpz\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-kube-api-access-r9tpz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.583779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583594    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.583779 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.583606    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:26.616035 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.616005    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:26.616216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.616091    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:58:26.684982 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.684945    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.685140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.685008    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.685140 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.685111    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwzw\" (UniqueName: \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.786477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.786371    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.786477 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.786464    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.786729 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.786530    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwzw\" (UniqueName: \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.787174 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.787152    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.788720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.788702    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.806970 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.806943    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwzw\" (UniqueName: \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\") pod \"maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.856870 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.856834    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:26.964561 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.964523    7037 generic.go:356] "Generic (PLEG): container finished" podID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" containerID="d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230" exitCode=0
Oct 30 21:58:26.964723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.964597    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"
Oct 30 21:58:26.964723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.964617    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" event={"ID":"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d","Type":"ContainerDied","Data":"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"}
Oct 30 21:58:26.964723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.964675    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g" event={"ID":"ef0f73ea-b35a-4aa1-952e-c21b7d09c25d","Type":"ContainerDied","Data":"e1c4b299ab088fb1821cb777cfef17da82603c4fd84bf2d5eff9fc9c935760d6"}
Oct 30 21:58:26.964723 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.964702    7037 scope.go:122] "RemoveContainer" containerID="d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"
Oct 30 21:58:26.977057 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.977036    7037 scope.go:122] "RemoveContainer" containerID="d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"
Oct 30 21:58:26.977373 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:26.977348    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230\": container with ID starting with d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230 not found: ID does not exist" containerID="d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"
Oct 30 21:58:26.977471 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:26.977380    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230"} err="failed to get container status \"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230\": rpc error: code = NotFound desc = could not find container \"d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230\": container with ID starting with d03920a342bfebdc8de81b92cdc82799350410606dbbc766bebed149cca96230 not found: ID does not exist"
Oct 30 21:58:27.016074 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.016039    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:58:27.018392 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.018366    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-b8s5g"]
Oct 30 21:58:27.152947 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.152918    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"]
Oct 30 21:58:27.153820 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:58:27.153799    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155c22f_8170_4886_8a98_2ed6b95457a3.slice/crio-d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24 WatchSource:0}: Error finding container d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24: Status 404 returned error can't find the container with id d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24
Oct 30 21:58:27.972302 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.972265    7037 generic.go:356] "Generic (PLEG): container finished" podID="6155c22f-8170-4886-8a98-2ed6b95457a3" containerID="942b344308ea9fdf61db11353765aa97e51395f56c421351f41f125c7059c483" exitCode=0
Oct 30 21:58:27.972697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.972317    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2" event={"ID":"6155c22f-8170-4886-8a98-2ed6b95457a3","Type":"ContainerDied","Data":"942b344308ea9fdf61db11353765aa97e51395f56c421351f41f125c7059c483"}
Oct 30 21:58:27.972697 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:27.972381    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2" event={"ID":"6155c22f-8170-4886-8a98-2ed6b95457a3","Type":"ContainerStarted","Data":"d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24"}
Oct 30 21:58:28.705780 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:28.705747    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0f73ea-b35a-4aa1-952e-c21b7d09c25d" path="/var/lib/kubelet/pods/ef0f73ea-b35a-4aa1-952e-c21b7d09c25d/volumes"
Oct 30 21:58:28.784003 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:28.783964    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:28.784191 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:28.783972    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:29.116577 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.116552    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:29.209951 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.209918    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\") pod \"6155c22f-8170-4886-8a98-2ed6b95457a3\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") "
Oct 30 21:58:29.210153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.209966    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\") pod \"6155c22f-8170-4886-8a98-2ed6b95457a3\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") "
Oct 30 21:58:29.210153 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.210093    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\" (UniqueName: \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\") pod \"6155c22f-8170-4886-8a98-2ed6b95457a3\" (UID: \"6155c22f-8170-4886-8a98-2ed6b95457a3\") "
Oct 30 21:58:29.210365 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.210341    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca" pod "6155c22f-8170-4886-8a98-2ed6b95457a3" (UID: "6155c22f-8170-4886-8a98-2ed6b95457a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:29.210473 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.210457    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:29.212226 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.212194    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca" pod "6155c22f-8170-4886-8a98-2ed6b95457a3" (UID: "6155c22f-8170-4886-8a98-2ed6b95457a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:29.212324 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.212221    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw" pod "6155c22f-8170-4886-8a98-2ed6b95457a3" (UID: "6155c22f-8170-4886-8a98-2ed6b95457a3"). InnerVolumeSpecName "kube-api-access-vvwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:29.311257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.311171    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c22f-8170-4886-8a98-2ed6b95457a3-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:29.311257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.311201    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwzw\" (UniqueName: \"kubernetes.io/projected/6155c22f-8170-4886-8a98-2ed6b95457a3-kube-api-access-vvwzw\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:29.985888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.985856    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"
Oct 30 21:58:29.986091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.985855    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2" event={"ID":"6155c22f-8170-4886-8a98-2ed6b95457a3","Type":"ContainerDied","Data":"d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24"}
Oct 30 21:58:29.986091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:29.985986    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3fece2fad3417876d1e1a783d611021a635bdb04edc78fdaadbd964038def24"
Oct 30 21:58:33.672150 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:33.672110    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:58:36.953557 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:36.953523    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:58:36.985286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:36.985246    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:58:37.469421 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.469382    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"]
Oct 30 21:58:37.469882 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.469867    7037 cpu_manager.go:414] "containerMap: removing container" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:37.469932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.469883    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3db193-1d90-49e9-9013-59bcbf381888" containerName="istio-proxy"
Oct 30 21:58:37.469932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.469913    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6155c22f-8170-4886-8a98-2ed6b95457a3" containerName="revoke-keys"
Oct 30 21:58:37.469932 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.469922    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6155c22f-8170-4886-8a98-2ed6b95457a3" containerName="revoke-keys"
Oct 30 21:58:37.470112 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.470095    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6155c22f-8170-4886-8a98-2ed6b95457a3" containerName="revoke-keys"
Oct 30 21:58:37.473639 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.473618    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.493159 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.493133    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:37.505269 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.505239    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:37.505427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.505411    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:37.505533 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.505475    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:58:37.507163 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.507141    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"]
Oct 30 21:58:37.592665 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.592633    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.592847 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.592685    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.592896 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.592850    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv725\" (UniqueName: \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.694216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.694181    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.694381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.694237    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.694381 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.694312    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv725\" (UniqueName: \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.694901 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.694876    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.696581 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.696560    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.713371 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.713347    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv725\" (UniqueName: \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\") pod \"maas-api-revoke-sub-ab6bf0d5-4j4rm\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:37.787822 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:37.787726    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:38.087266 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:38.087233    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"]
Oct 30 21:58:38.087784 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:58:38.087762    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ebe198_0946_49cf_bd24_5be387389455.slice/crio-5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7 WatchSource:0}: Error finding container 5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7: Status 404 returned error can't find the container with id 5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7
Oct 30 21:58:38.783808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:38.783715    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:38.783808 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:38.783728    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:39.033690 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:39.033650    7037 generic.go:356] "Generic (PLEG): container finished" podID="83ebe198-0946-49cf-bd24-5be387389455" containerID="d80ac0e216441dce653df06122088f462ce1b3e86b96173c545c20f98bea455f" exitCode=0
Oct 30 21:58:39.033945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:39.033739    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm" event={"ID":"83ebe198-0946-49cf-bd24-5be387389455","Type":"ContainerDied","Data":"d80ac0e216441dce653df06122088f462ce1b3e86b96173c545c20f98bea455f"}
Oct 30 21:58:39.033945 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:39.033795    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm" event={"ID":"83ebe198-0946-49cf-bd24-5be387389455","Type":"ContainerStarted","Data":"5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7"}
Oct 30 21:58:40.232100 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.232068    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:40.323257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.323222    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\" (UniqueName: \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\") pod \"83ebe198-0946-49cf-bd24-5be387389455\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") "
Oct 30 21:58:40.323257 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.323260    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\") pod \"83ebe198-0946-49cf-bd24-5be387389455\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") "
Oct 30 21:58:40.323512 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.323359    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\") pod \"83ebe198-0946-49cf-bd24-5be387389455\" (UID: \"83ebe198-0946-49cf-bd24-5be387389455\") "
Oct 30 21:58:40.323809 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.323767    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca" pod "83ebe198-0946-49cf-bd24-5be387389455" (UID: "83ebe198-0946-49cf-bd24-5be387389455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:40.325589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.325544    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca" pod "83ebe198-0946-49cf-bd24-5be387389455" (UID: "83ebe198-0946-49cf-bd24-5be387389455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:40.325589 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.325581    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725" pod "83ebe198-0946-49cf-bd24-5be387389455" (UID: "83ebe198-0946-49cf-bd24-5be387389455"). InnerVolumeSpecName "kube-api-access-pv725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:40.424713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.424671    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ebe198-0946-49cf-bd24-5be387389455-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:40.424713 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.424710    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv725\" (UniqueName: \"kubernetes.io/projected/83ebe198-0946-49cf-bd24-5be387389455-kube-api-access-pv725\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:40.424918 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:40.424725    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ebe198-0946-49cf-bd24-5be387389455-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:41.074029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:41.073992    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"
Oct 30 21:58:41.074029 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:41.074015    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm" event={"ID":"83ebe198-0946-49cf-bd24-5be387389455","Type":"ContainerDied","Data":"5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7"}
Oct 30 21:58:41.074244 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:41.074069    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5072216124e814108e84a5352803292d186bdd520e1856a902bd03b8940b43a7"
Oct 30 21:58:41.985994 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:41.985948    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing" containerID="cri-o://6e30ad32ced8070eb4b06d9740b6b7b0d49e7903fc605a49e931c582fa349555" gracePeriod=25
Oct 30 21:58:42.020650 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.020609    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing" containerID="cri-o://4fd0b515b04da79e8640e512b811189578b3bdba43550a1fd667e6b45372d508" gracePeriod=25
Oct 30 21:58:42.412877 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.412844    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:58:42.434230 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.434208    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:58:42.444732 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.444710    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\" (UniqueName: \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\") pod \"08137a39-98c2-4895-afb4-6cfcccc27932\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") "
Oct 30 21:58:42.444823 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.444763    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\") pod \"08137a39-98c2-4895-afb4-6cfcccc27932\" (UID: \"08137a39-98c2-4895-afb4-6cfcccc27932\") "
Oct 30 21:58:42.444888 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.444866    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\") pod \"265bdf6b-1fee-496e-ba32-a516560d5fb1\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") "
Oct 30 21:58:42.444939 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.444892    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\" (UniqueName: \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\") pod \"265bdf6b-1fee-496e-ba32-a516560d5fb1\" (UID: \"265bdf6b-1fee-496e-ba32-a516560d5fb1\") "
Oct 30 21:58:42.445249 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.445221    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume" pod "08137a39-98c2-4895-afb4-6cfcccc27932" (UID: "08137a39-98c2-4895-afb4-6cfcccc27932"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:42.445332 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.445250    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume" pod "265bdf6b-1fee-496e-ba32-a516560d5fb1" (UID: "265bdf6b-1fee-496e-ba32-a516560d5fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:42.446832 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.446811    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd" pod "08137a39-98c2-4895-afb4-6cfcccc27932" (UID: "08137a39-98c2-4895-afb4-6cfcccc27932"). InnerVolumeSpecName "kube-api-access-68hsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:42.446912 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.446878    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx" pod "265bdf6b-1fee-496e-ba32-a516560d5fb1" (UID: "265bdf6b-1fee-496e-ba32-a516560d5fb1"). InnerVolumeSpecName "kube-api-access-xhvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:42.545667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.545576    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hsd\" (UniqueName: \"kubernetes.io/projected/08137a39-98c2-4895-afb4-6cfcccc27932-kube-api-access-68hsd\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:42.545667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.545607    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08137a39-98c2-4895-afb4-6cfcccc27932-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:42.545667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.545617    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265bdf6b-1fee-496e-ba32-a516560d5fb1-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:42.545667 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:42.545625    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhvrx\" (UniqueName: \"kubernetes.io/projected/265bdf6b-1fee-496e-ba32-a516560d5fb1-kube-api-access-xhvrx\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:58:43.085720 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.085685    7037 generic.go:356] "Generic (PLEG): container finished" podID="08137a39-98c2-4895-afb4-6cfcccc27932" containerID="c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9" exitCode=0
Oct 30 21:58:43.086204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.085733    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"
Oct 30 21:58:43.086204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.085762    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" event={"ID":"08137a39-98c2-4895-afb4-6cfcccc27932","Type":"ContainerDied","Data":"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"}
Oct 30 21:58:43.086204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.085820    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w" event={"ID":"08137a39-98c2-4895-afb4-6cfcccc27932","Type":"ContainerDied","Data":"2f08c8c12411639f1b60c3a268593dde8265592e484531fac67c80eb6615ed58"}
Oct 30 21:58:43.086204 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.085849    7037 scope.go:122] "RemoveContainer" containerID="c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"
Oct 30 21:58:43.087212 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.087164    7037 generic.go:356] "Generic (PLEG): container finished" podID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerID="a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838" exitCode=0
Oct 30 21:58:43.087286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.087209    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"
Oct 30 21:58:43.087286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.087225    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" event={"ID":"265bdf6b-1fee-496e-ba32-a516560d5fb1","Type":"ContainerDied","Data":"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"}
Oct 30 21:58:43.087286 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.087269    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p" event={"ID":"265bdf6b-1fee-496e-ba32-a516560d5fb1","Type":"ContainerDied","Data":"bbf8c940a5813ded507804a19407c827308b4758c1aab0a1094e67c506f18156"}
Oct 30 21:58:43.098369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.098348    7037 scope.go:122] "RemoveContainer" containerID="c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"
Oct 30 21:58:43.098701 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:43.098668    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9\": container with ID starting with c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9 not found: ID does not exist" containerID="c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"
Oct 30 21:58:43.098813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.098705    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9"} err="failed to get container status \"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9\": rpc error: code = NotFound desc = could not find container \"c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9\": container with ID starting with c3aa75fbcce7ec9d38329a54140c76ca80b5ec30764843e5c2287b4eb4eedfa9 not found: ID does not exist"
Oct 30 21:58:43.098813 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.098723    7037 scope.go:122] "RemoveContainer" containerID="a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"
Oct 30 21:58:43.107642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.107624    7037 scope.go:122] "RemoveContainer" containerID="a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"
Oct 30 21:58:43.107922 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:58:43.107904    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838\": container with ID starting with a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838 not found: ID does not exist" containerID="a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"
Oct 30 21:58:43.107984 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.107929    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838"} err="failed to get container status \"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838\": rpc error: code = NotFound desc = could not find container \"a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838\": container with ID starting with a281bdc8c075b1b1c5a65446f2a906203c82c5ce75ee8aa6f9c7b6c5a14ae838 not found: ID does not exist"
Oct 30 21:58:43.137573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.137532    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:58:43.149139 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.149099    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-95692f-749d9cd959-9jm9p"]
Oct 30 21:58:43.175080 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.175039    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:58:43.177506 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:43.177469    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-95692f-7697cf9c9b-pvc4w"]
Oct 30 21:58:44.705610 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:44.705575    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" path="/var/lib/kubelet/pods/08137a39-98c2-4895-afb4-6cfcccc27932/volumes"
Oct 30 21:58:44.706015 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:44.706000    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" path="/var/lib/kubelet/pods/265bdf6b-1fee-496e-ba32-a516560d5fb1/volumes"
Oct 30 21:58:48.336955 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:48.336915    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:49.340214 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:49.340174    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:53.138848 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:53.138810    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"]
Oct 30 21:58:53.150837 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:53.150807    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ed088d-w5kb2"]
Oct 30 21:58:54.705479 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:54.705443    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6155c22f-8170-4886-8a98-2ed6b95457a3" path="/var/lib/kubelet/pods/6155c22f-8170-4886-8a98-2ed6b95457a3/volumes"
Oct 30 21:58:58.336369 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:58.336330    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:59.340632 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:58:59.340594    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:59:00.018432 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:00.018381    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:59:00.018688 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:00.018662    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" containerID="cri-o://376e9171ee736e09854c323716fdc6c30aed837c5cb5b2c5f43a8f55a670357e" gracePeriod=30
Oct 30 21:59:01.816686 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:01.816650    7037 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:59:01.817104 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:01.816715    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:59:02.196024 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.195983    7037 generic.go:356] "Generic (PLEG): container finished" podID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerID="6e30ad32ced8070eb4b06d9740b6b7b0d49e7903fc605a49e931c582fa349555" exitCode=0
Oct 30 21:59:02.196334 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.196105    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" event={"ID":"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a","Type":"ContainerDied","Data":"6e30ad32ced8070eb4b06d9740b6b7b0d49e7903fc605a49e931c582fa349555"}
Oct 30 21:59:02.197962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.197835    7037 generic.go:356] "Generic (PLEG): container finished" podID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerID="4fd0b515b04da79e8640e512b811189578b3bdba43550a1fd667e6b45372d508" exitCode=0
Oct 30 21:59:02.197962 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.197910    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" event={"ID":"c85b6942-19ff-4eb5-8ce3-46795a93ef34","Type":"ContainerDied","Data":"4fd0b515b04da79e8640e512b811189578b3bdba43550a1fd667e6b45372d508"}
Oct 30 21:59:02.304997 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.304969    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:59:02.308043 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.308018    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:59:02.457136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457046    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\") pod \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") "
Oct 30 21:59:02.457136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457090    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\" (UniqueName: \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\") pod \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") "
Oct 30 21:59:02.457136 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457117    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\" (UniqueName: \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\") pod \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\" (UID: \"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a\") "
Oct 30 21:59:02.457427 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457201    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\") pod \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\" (UID: \"c85b6942-19ff-4eb5-8ce3-46795a93ef34\") "
Oct 30 21:59:02.457587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457548    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume" pod "b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" (UID: "b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:02.457587 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.457561    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume" pod "c85b6942-19ff-4eb5-8ce3-46795a93ef34" (UID: "c85b6942-19ff-4eb5-8ce3-46795a93ef34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:02.459273 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.459255    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45" pod "b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" (UID: "b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a"). InnerVolumeSpecName "kube-api-access-mkv45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:02.459331 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.459312    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9" pod "c85b6942-19ff-4eb5-8ce3-46795a93ef34" (UID: "c85b6942-19ff-4eb5-8ce3-46795a93ef34"). InnerVolumeSpecName "kube-api-access-mnzt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:02.559008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.558958    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:02.559008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.559000    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzt9\" (UniqueName: \"kubernetes.io/projected/c85b6942-19ff-4eb5-8ce3-46795a93ef34-kube-api-access-mnzt9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:02.559008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.559010    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkv45\" (UniqueName: \"kubernetes.io/projected/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a-kube-api-access-mkv45\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:02.559008 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:02.559020    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b6942-19ff-4eb5-8ce3-46795a93ef34-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:03.204818 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.204786    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"
Oct 30 21:59:03.205216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.204796    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j" event={"ID":"b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a","Type":"ContainerDied","Data":"f733bdff1e7d98a701106e4155d1a3527c0bdccdf462e2da77de9023be267e6e"}
Oct 30 21:59:03.205216 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.204926    7037 scope.go:122] "RemoveContainer" containerID="6e30ad32ced8070eb4b06d9740b6b7b0d49e7903fc605a49e931c582fa349555"
Oct 30 21:59:03.206859 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.206837    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h" event={"ID":"c85b6942-19ff-4eb5-8ce3-46795a93ef34","Type":"ContainerDied","Data":"9593b9ecdc0d2bfe56b7bc38968ffbb8ae9c673b389743557f8380d4ed3dca40"}
Oct 30 21:59:03.207019 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.206929    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"
Oct 30 21:59:03.217922 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.217897    7037 scope.go:122] "RemoveContainer" containerID="4fd0b515b04da79e8640e512b811189578b3bdba43550a1fd667e6b45372d508"
Oct 30 21:59:03.255633 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.255570    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:59:03.259768 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.259746    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ed088d-dfb468776-kmn7h"]
Oct 30 21:59:03.293250 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.293222    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:59:03.297033 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:03.297006    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ed088d-5c4d48b955-p2s5j"]
Oct 30 21:59:03.671482 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:59:03.671434    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 21:59:04.705466 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:04.705430    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" path="/var/lib/kubelet/pods/b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a/volumes"
Oct 30 21:59:04.705911 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:04.705864    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" path="/var/lib/kubelet/pods/c85b6942-19ff-4eb5-8ce3-46795a93ef34/volumes"
Oct 30 21:59:05.221285 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.221253    7037 generic.go:356] "Generic (PLEG): container finished" podID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerID="376e9171ee736e09854c323716fdc6c30aed837c5cb5b2c5f43a8f55a670357e" exitCode=0
Oct 30 21:59:05.221510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.221335    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerDied","Data":"376e9171ee736e09854c323716fdc6c30aed837c5cb5b2c5f43a8f55a670357e"}
Oct 30 21:59:05.221510 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.221418    7037 scope.go:122] "RemoveContainer" containerID="21e14e7ec1d004244d3e73633e4e4c3eb8f7d849dee84fbe7773f79718c3bb42"
Oct 30 21:59:05.279543 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.279513    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:59:05.385454 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385420    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385543    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385575    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385622    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385660 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385650    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385681    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385730    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385763    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.385872 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385795    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\") pod \"1c470506-029a-404a-9441-7f7cd07dd3fc\" (UID: \"1c470506-029a-404a-9441-7f7cd07dd3fc\") "
Oct 30 21:59:05.386091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.385927    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:59:05.386091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386022    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:05.386091 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386032    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:59:05.386254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386170    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:59:05.386254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386226    7037 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c470506-029a-404a-9441-7f7cd07dd3fc-istiod-ca-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.386254 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386246    7037 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.386401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386260    7037 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-credential-socket\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.386401 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.386380    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:59:05.387886 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.387861    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "kube-api-access-wst22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:05.388272 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.388255    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:05.388343 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.388284    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:59:05.389215 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.389197    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy" pod "1c470506-029a-404a-9441-7f7cd07dd3fc" (UID: "1c470506-029a-404a-9441-7f7cd07dd3fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:59:05.487265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487222    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.487265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487256    7037 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-data\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.487265 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487269    7037 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-podinfo\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.487573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487280    7037 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-workload-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.487573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487293    7037 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c470506-029a-404a-9441-7f7cd07dd3fc-istio-envoy\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:05.487573 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:05.487304    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wst22\" (UniqueName: \"kubernetes.io/projected/1c470506-029a-404a-9441-7f7cd07dd3fc-kube-api-access-wst22\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:06.227201 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.227171    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"
Oct 30 21:59:06.227642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.227213    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz" event={"ID":"1c470506-029a-404a-9441-7f7cd07dd3fc","Type":"ContainerDied","Data":"853febaf1975584c9892d58bfa488f827a7aa45ee74fc2557dadd731791ebeee"}
Oct 30 21:59:06.227642 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.227273    7037 scope.go:122] "RemoveContainer" containerID="376e9171ee736e09854c323716fdc6c30aed837c5cb5b2c5f43a8f55a670357e"
Oct 30 21:59:06.290852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.290808    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:59:06.301349 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.301313    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-wdzbz"]
Oct 30 21:59:06.705891 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:06.705858    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" path="/var/lib/kubelet/pods/1c470506-029a-404a-9441-7f7cd07dd3fc/volumes"
Oct 30 21:59:26.141550 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.141508    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"]
Oct 30 21:59:26.142127 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142104    7037 cpu_manager.go:414] "containerMap: removing container" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142129    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142154    7037 cpu_manager.go:414] "containerMap: removing container" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142163    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142178    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142186    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142206    7037 cpu_manager.go:414] "containerMap: removing container" podUID="83ebe198-0946-49cf-bd24-5be387389455" containerName="revoke-keys"
Oct 30 21:59:26.142219 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142214    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ebe198-0946-49cf-bd24-5be387389455" containerName="revoke-keys"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142231    7037 cpu_manager.go:414] "containerMap: removing container" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142239    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142253    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142261    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142286    7037 cpu_manager.go:414] "containerMap: removing container" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142295    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142403    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ebe198-0946-49cf-bd24-5be387389455" containerName="revoke-keys"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142419    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85b6942-19ff-4eb5-8ce3-46795a93ef34" containerName="payload-processing"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142429    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142443    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c470506-029a-404a-9441-7f7cd07dd3fc" containerName="istio-proxy"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142457    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="08137a39-98c2-4895-afb4-6cfcccc27932" containerName="payload-processing"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142468    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="265bdf6b-1fee-496e-ba32-a516560d5fb1" containerName="payload-pre-processing"
Oct 30 21:59:26.142634 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.142479    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dbfe4e-34b9-48a0-a9e7-f4b349d1271a" containerName="payload-pre-processing"
Oct 30 21:59:26.147669 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.147647    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.163673 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.163646    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:26.163868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.163734    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:59:26.163868 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.163810    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"]
Oct 30 21:59:26.174458 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.174433    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:59:26.174705 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.174688    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 21:59:26.292037 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.292003    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.292190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.292049    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.292190 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.292149    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxr5\" (UniqueName: \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.393651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.393558    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.393651 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.393629    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxr5\" (UniqueName: \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.393885 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.393698    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.394225 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.394199    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.396055 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.396034    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.415110 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.415087    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxr5\" (UniqueName: \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\") pod \"maas-api-revoke-sub-85237e45-qcbdm\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.460109 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.460079    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:26.752860 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:26.752834    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"]
Oct 30 21:59:26.754637 ip-10-0-137-72 kubenswrapper[7037]: W1030 21:59:26.754610    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad08b54f_4278_4e8b_917c_56e64abeb0d2.slice/crio-1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3 WatchSource:0}: Error finding container 1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3: Status 404 returned error can't find the container with id 1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3
Oct 30 21:59:27.334703 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:27.334662    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm" event={"ID":"ad08b54f-4278-4e8b-917c-56e64abeb0d2","Type":"ContainerStarted","Data":"3699a64d3ed2644be43df5c588231c2724af0e7a4332d59c3493c344045fbb03"}
Oct 30 21:59:27.335094 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:27.334726    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm" event={"ID":"ad08b54f-4278-4e8b-917c-56e64abeb0d2","Type":"ContainerStarted","Data":"1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3"}
Oct 30 21:59:27.376060 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:27.375998    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm" podStartSLOduration=1.3759838229999999 podStartE2EDuration="1.375983823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:27.37437248 +0000 UTC m=+3510.785911916" watchObservedRunningTime="2026-10-30 21:59:27.375983823 +0000 UTC m=+3510.787523275"
Oct 30 21:59:28.341222 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:28.341187    7037 generic.go:356] "Generic (PLEG): container finished" podID="ad08b54f-4278-4e8b-917c-56e64abeb0d2" containerID="3699a64d3ed2644be43df5c588231c2724af0e7a4332d59c3493c344045fbb03" exitCode=0
Oct 30 21:59:28.341683 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:28.341248    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm" event={"ID":"ad08b54f-4278-4e8b-917c-56e64abeb0d2","Type":"ContainerDied","Data":"3699a64d3ed2644be43df5c588231c2724af0e7a4332d59c3493c344045fbb03"}
Oct 30 21:59:29.483119 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.483096    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:29.624268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.624183    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\" (UniqueName: \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\") pod \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") "
Oct 30 21:59:29.624268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.624221    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\") pod \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") "
Oct 30 21:59:29.624268 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.624262    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\") pod \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\" (UID: \"ad08b54f-4278-4e8b-917c-56e64abeb0d2\") "
Oct 30 21:59:29.624737 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.624649    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca" pod "ad08b54f-4278-4e8b-917c-56e64abeb0d2" (UID: "ad08b54f-4278-4e8b-917c-56e64abeb0d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:29.626556 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.626527    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5" pod "ad08b54f-4278-4e8b-917c-56e64abeb0d2" (UID: "ad08b54f-4278-4e8b-917c-56e64abeb0d2"). InnerVolumeSpecName "kube-api-access-qbxr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:29.626680 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.626532    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca" pod "ad08b54f-4278-4e8b-917c-56e64abeb0d2" (UID: "ad08b54f-4278-4e8b-917c-56e64abeb0d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:29.725852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.725815    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxr5\" (UniqueName: \"kubernetes.io/projected/ad08b54f-4278-4e8b-917c-56e64abeb0d2-kube-api-access-qbxr5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:29.725852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.725849    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:29.725852 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:29.725860    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad08b54f-4278-4e8b-917c-56e64abeb0d2-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 21:59:30.352728 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:30.352695    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"
Oct 30 21:59:30.352946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:30.352700    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm" event={"ID":"ad08b54f-4278-4e8b-917c-56e64abeb0d2","Type":"ContainerDied","Data":"1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3"}
Oct 30 21:59:30.352946 ip-10-0-137-72 kubenswrapper[7037]: I1030 21:59:30.352822    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4e96bf43ee7ace3b473361311860f8f0a3d68aa67264aefe382a711407b9c3"
Oct 30 21:59:33.671780 ip-10-0-137-72 kubenswrapper[7037]: E1030 21:59:33.671734    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:00:00.146990 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.146950    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:00.147430 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.147376    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ad08b54f-4278-4e8b-917c-56e64abeb0d2" containerName="revoke-keys"
Oct 30 22:00:00.147430 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.147390    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad08b54f-4278-4e8b-917c-56e64abeb0d2" containerName="revoke-keys"
Oct 30 22:00:00.147531 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.147484    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad08b54f-4278-4e8b-917c-56e64abeb0d2" containerName="revoke-keys"
Oct 30 22:00:00.152104 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.152083    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.167520 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.167438    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.167520 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.167454    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.169359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.169330    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:00.178972 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.178935    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:00:00.234238 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.234198    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.234238 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.234242    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tkr\" (UniqueName: \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.335134 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.335097    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.335134 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.335139    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tkr\" (UniqueName: \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.335795 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.335774    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.356634 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.356600    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tkr\" (UniqueName: \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\") pod \"maas-api-key-cleanup-29889960-n68gd\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.466349 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.466265    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:00.768402 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:00:00.768361    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea47e5cc_02a9_458a_9341_9cf41950f29f.slice/crio-c9fe8c33cad25ac5c24ee57b0184a8ef8dd68e8eb195a346cbedb49bb4dcf753 WatchSource:0}: Error finding container c9fe8c33cad25ac5c24ee57b0184a8ef8dd68e8eb195a346cbedb49bb4dcf753: Status 404 returned error can't find the container with id c9fe8c33cad25ac5c24ee57b0184a8ef8dd68e8eb195a346cbedb49bb4dcf753
Oct 30 22:00:00.768556 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:00.768432    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:01.503596 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:01.503558    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerStarted","Data":"923161039eb54a25b43d92b44eb43c5beb3c478dce3baa3a08285f92e61fda64"}
Oct 30 22:00:01.504120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:01.503618    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerStarted","Data":"c9fe8c33cad25ac5c24ee57b0184a8ef8dd68e8eb195a346cbedb49bb4dcf753"}
Oct 30 22:00:01.545441 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:01.545385    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" podStartSLOduration=1.54537213 podStartE2EDuration="1.54537213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.544170684 +0000 UTC m=+3544.955710122" watchObservedRunningTime="2026-10-30 22:00:01.54537213 +0000 UTC m=+3544.956911610"
Oct 30 22:00:02.511311 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:02.511278    7037 generic.go:356] "Generic (PLEG): container finished" podID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerID="923161039eb54a25b43d92b44eb43c5beb3c478dce3baa3a08285f92e61fda64" exitCode=60
Oct 30 22:00:02.511837 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:02.511361    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerDied","Data":"923161039eb54a25b43d92b44eb43c5beb3c478dce3baa3a08285f92e61fda64"}
Oct 30 22:00:02.511837 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:02.511656    7037 scope.go:122] "RemoveContainer" containerID="923161039eb54a25b43d92b44eb43c5beb3c478dce3baa3a08285f92e61fda64"
Oct 30 22:00:03.517735 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:03.517704    7037 generic.go:356] "Generic (PLEG): container finished" podID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerID="69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5" exitCode=60
Oct 30 22:00:03.518197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:03.517792    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerDied","Data":"69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5"}
Oct 30 22:00:03.518197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:03.517867    7037 scope.go:122] "RemoveContainer" containerID="923161039eb54a25b43d92b44eb43c5beb3c478dce3baa3a08285f92e61fda64"
Oct 30 22:00:03.518197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:03.518115    7037 scope.go:122] "RemoveContainer" containerID="69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5"
Oct 30 22:00:03.518382 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:03.518364    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-n68gd_odh-ai-gateway-infra(ea47e5cc-02a9-458a-9341-9cf41950f29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f"
Oct 30 22:00:03.671317 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:03.671227    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:00:04.524203 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:04.524176    7037 scope.go:122] "RemoveContainer" containerID="69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5"
Oct 30 22:00:04.524633 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:04.524388    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-n68gd_odh-ai-gateway-infra(ea47e5cc-02a9-458a-9341-9cf41950f29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f"
Oct 30 22:00:07.508187 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.508152    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 22:00:07.508783 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.508339    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" podUID="b443c850-ad96-445e-9e45-538af32948b0" containerName="discovery" containerID="cri-o://4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6" gracePeriod=30
Oct 30 22:00:07.776005 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.775979    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 22:00:07.909342 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909309    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909355    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909386    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909419    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909471    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909819 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909579    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909819 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909643    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\") pod \"b443c850-ad96-445e-9e45-538af32948b0\" (UID: \"b443c850-ad96-445e-9e45-538af32948b0\") "
Oct 30 22:00:07.909949 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.909920    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:07.910082 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.910050    7037 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-ca-configmap\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:07.911853 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.911816    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:07.912008 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.911942    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:07.912167 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.912144    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "kube-api-access-fgqxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:07.912167 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.912160    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:07.912311 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.912182    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:07.912311 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:07.912200    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts" pod "b443c850-ad96-445e-9e45-538af32948b0" (UID: "b443c850-ad96-445e-9e45-538af32948b0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.010948    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqxh\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-kube-api-access-fgqxh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.010982    7037 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b443c850-ad96-445e-9e45-538af32948b0-local-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.010996    7037 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-kubeconfig\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.011007    7037 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-istio-csr-dns-cert\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.011018    7037 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b443c850-ad96-445e-9e45-538af32948b0-istio-token\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.011040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.011029    7037 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b443c850-ad96-445e-9e45-538af32948b0-cacerts\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.546444 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.546408    7037 generic.go:356] "Generic (PLEG): container finished" podID="b443c850-ad96-445e-9e45-538af32948b0" containerID="4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6" exitCode=0
Oct 30 22:00:08.546867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.546459    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"
Oct 30 22:00:08.546867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.546465    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" event={"ID":"b443c850-ad96-445e-9e45-538af32948b0","Type":"ContainerDied","Data":"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"}
Oct 30 22:00:08.546867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.546534    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j" event={"ID":"b443c850-ad96-445e-9e45-538af32948b0","Type":"ContainerDied","Data":"8ea975b059a8293a83460a6d7da4c0f809bf8b0c7afb7c417a60d162e6d31377"}
Oct 30 22:00:08.546867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.546562    7037 scope.go:122] "RemoveContainer" containerID="4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"
Oct 30 22:00:08.556057 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.556022    7037 scope.go:122] "RemoveContainer" containerID="4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"
Oct 30 22:00:08.556360 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:08.556340    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6\": container with ID starting with 4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6 not found: ID does not exist" containerID="4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"
Oct 30 22:00:08.556418 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.556369    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6"} err="failed to get container status \"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6\": rpc error: code = NotFound desc = could not find container \"4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6\": container with ID starting with 4897f7a80c613d0fc72b9b2833fa67278a3e71479d32c32f1b8edb35ce5cadd6 not found: ID does not exist"
Oct 30 22:00:08.611380 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.611343    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 22:00:08.620735 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.620705    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzl4j"]
Oct 30 22:00:08.705756 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:08.705721    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b443c850-ad96-445e-9e45-538af32948b0" path="/var/lib/kubelet/pods/b443c850-ad96-445e-9e45-538af32948b0/volumes"
Oct 30 22:00:16.702148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:16.702119    7037 scope.go:122] "RemoveContainer" containerID="69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5"
Oct 30 22:00:17.593185 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:17.593150    7037 generic.go:356] "Generic (PLEG): container finished" podID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerID="bfff62501432e3e1eaecce2ffe019bd5fa2a0c139111ecc8d21b0c304fb0ca67" exitCode=60
Oct 30 22:00:17.593396 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:17.593221    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerDied","Data":"bfff62501432e3e1eaecce2ffe019bd5fa2a0c139111ecc8d21b0c304fb0ca67"}
Oct 30 22:00:17.593396 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:17.593287    7037 scope.go:122] "RemoveContainer" containerID="69b966ea94ad0b6fa3fbb21dc809d9ef3b94940cc107c52ec5d93465195c70d5"
Oct 30 22:00:17.593698 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:17.593677    7037 scope.go:122] "RemoveContainer" containerID="bfff62501432e3e1eaecce2ffe019bd5fa2a0c139111ecc8d21b0c304fb0ca67"
Oct 30 22:00:17.593938 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:17.593916    7037 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-n68gd_odh-ai-gateway-infra(ea47e5cc-02a9-458a-9341-9cf41950f29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f"
Oct 30 22:00:18.642482 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.642445    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:18.775153 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.775126    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:18.905281 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.905200    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\" (UniqueName: \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\") pod \"ea47e5cc-02a9-458a-9341-9cf41950f29f\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") "
Oct 30 22:00:18.905428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.905290    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\") pod \"ea47e5cc-02a9-458a-9341-9cf41950f29f\" (UID: \"ea47e5cc-02a9-458a-9341-9cf41950f29f\") "
Oct 30 22:00:18.905769 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.905739    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca" pod "ea47e5cc-02a9-458a-9341-9cf41950f29f" (UID: "ea47e5cc-02a9-458a-9341-9cf41950f29f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:18.907323 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:18.907298    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr" pod "ea47e5cc-02a9-458a-9341-9cf41950f29f" (UID: "ea47e5cc-02a9-458a-9341-9cf41950f29f"). InnerVolumeSpecName "kube-api-access-z4tkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.006829 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.006787    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tkr\" (UniqueName: \"kubernetes.io/projected/ea47e5cc-02a9-458a-9341-9cf41950f29f-kube-api-access-z4tkr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.006829 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.006821    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea47e5cc-02a9-458a-9341-9cf41950f29f-maas-api-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.605602 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.605568    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd" event={"ID":"ea47e5cc-02a9-458a-9341-9cf41950f29f","Type":"ContainerDied","Data":"c9fe8c33cad25ac5c24ee57b0184a8ef8dd68e8eb195a346cbedb49bb4dcf753"}
Oct 30 22:00:19.605791 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.605632    7037 scope.go:122] "RemoveContainer" containerID="bfff62501432e3e1eaecce2ffe019bd5fa2a0c139111ecc8d21b0c304fb0ca67"
Oct 30 22:00:19.605791 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.605579    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"
Oct 30 22:00:19.656826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.656796    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:19.660044 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:19.660017    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-n68gd"]
Oct 30 22:00:20.706048 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:20.706018    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" path="/var/lib/kubelet/pods/ea47e5cc-02a9-458a-9341-9cf41950f29f/volumes"
Oct 30 22:00:22.824258 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824221    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"]
Oct 30 22:00:22.824808 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824791    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b443c850-ad96-445e-9e45-538af32948b0" containerName="discovery"
Oct 30 22:00:22.824856 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824810    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b443c850-ad96-445e-9e45-538af32948b0" containerName="discovery"
Oct 30 22:00:22.824856 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824833    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.824856 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824841    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.824959 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824893    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.824959 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824902    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.825025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824978    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.825025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.824990    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b443c850-ad96-445e-9e45-538af32948b0" containerName="discovery"
Oct 30 22:00:22.825025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.825002    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.825025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.825012    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:22.829896 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.829877    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:22.845761 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.845734    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"]
Oct 30 22:00:22.856074 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.856048    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:22.856215 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.856094    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:22.856215 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.856118    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:00:22.857122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.857108    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:22.946090 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.946058    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:22.946276 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.946103    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:22.946276 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:22.946235    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxbs\" (UniqueName: \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.046939 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.046904    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxbs\" (UniqueName: \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.047120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.047007    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.047120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.047049    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.047710 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.047689    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.049580 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.049546    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.066681 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.066659    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxbs\" (UniqueName: \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\") pod \"maas-api-revoke-sub-c1e271a9-dwdww\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.142606 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.142513    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:23.432888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.432863    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"]
Oct 30 22:00:23.434352 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:00:23.434321    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7635891c_b3c1_4f2e_b830_a57682c03eab.slice/crio-a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f WatchSource:0}: Error finding container a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f: Status 404 returned error can't find the container with id a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f
Oct 30 22:00:23.633819 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:23.633783    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww" event={"ID":"7635891c-b3c1-4f2e-b830-a57682c03eab","Type":"ContainerStarted","Data":"a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f"}
Oct 30 22:00:24.640384 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:24.640347    7037 generic.go:356] "Generic (PLEG): container finished" podID="7635891c-b3c1-4f2e-b830-a57682c03eab" containerID="e2b53d1ecf07ef2782e1c7102313ea0a1fcfee0446143e6e86ce1ff3263907d6" exitCode=0
Oct 30 22:00:24.640898 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:24.640434    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww" event={"ID":"7635891c-b3c1-4f2e-b830-a57682c03eab","Type":"ContainerDied","Data":"e2b53d1ecf07ef2782e1c7102313ea0a1fcfee0446143e6e86ce1ff3263907d6"}
Oct 30 22:00:25.778411 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.778385    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:25.876151 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.876121    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\" (UniqueName: \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\") pod \"7635891c-b3c1-4f2e-b830-a57682c03eab\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") "
Oct 30 22:00:25.876359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.876200    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\") pod \"7635891c-b3c1-4f2e-b830-a57682c03eab\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") "
Oct 30 22:00:25.876359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.876262    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\") pod \"7635891c-b3c1-4f2e-b830-a57682c03eab\" (UID: \"7635891c-b3c1-4f2e-b830-a57682c03eab\") "
Oct 30 22:00:25.876664 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.876637    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca" pod "7635891c-b3c1-4f2e-b830-a57682c03eab" (UID: "7635891c-b3c1-4f2e-b830-a57682c03eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:25.878321 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.878292    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs" pod "7635891c-b3c1-4f2e-b830-a57682c03eab" (UID: "7635891c-b3c1-4f2e-b830-a57682c03eab"). InnerVolumeSpecName "kube-api-access-hbxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:25.878507 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.878475    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca" pod "7635891c-b3c1-4f2e-b830-a57682c03eab" (UID: "7635891c-b3c1-4f2e-b830-a57682c03eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:25.977595 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.977481    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbxbs\" (UniqueName: \"kubernetes.io/projected/7635891c-b3c1-4f2e-b830-a57682c03eab-kube-api-access-hbxbs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:25.977595 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.977543    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:25.977595 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:25.977552    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7635891c-b3c1-4f2e-b830-a57682c03eab-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:26.651477 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:26.651444    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"
Oct 30 22:00:26.651684 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:26.651443    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww" event={"ID":"7635891c-b3c1-4f2e-b830-a57682c03eab","Type":"ContainerDied","Data":"a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f"}
Oct 30 22:00:26.651684 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:26.651558    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f096d6e4312636c2d267d110f8c05e5dddfc7aba124104e1cef1d81752809f"
Oct 30 22:00:33.671975 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:00:33.671940    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:00:41.990249 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.990210    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"]
Oct 30 22:00:41.990864 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.990734    7037 cpu_manager.go:414] "containerMap: removing container" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:41.990864 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.990752    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea47e5cc-02a9-458a-9341-9cf41950f29f" containerName="cleanup"
Oct 30 22:00:41.990864 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.990781    7037 cpu_manager.go:414] "containerMap: removing container" podUID="7635891c-b3c1-4f2e-b830-a57682c03eab" containerName="revoke-keys"
Oct 30 22:00:41.990864 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.990790    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7635891c-b3c1-4f2e-b830-a57682c03eab" containerName="revoke-keys"
Oct 30 22:00:41.991338 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.991037    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="7635891c-b3c1-4f2e-b830-a57682c03eab" containerName="revoke-keys"
Oct 30 22:00:41.994419 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:41.994399    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.009585 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.009562    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:00:42.012037 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012015    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szf88\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-kube-api-access-szf88\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012107 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012050    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012107 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012070    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012178    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012234 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012207    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538657b-f203-4b93-80d6-77522db77d56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012234 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012229    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012305 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012245    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012347 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012335    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538657b-f203-4b93-80d6-77522db77d56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.012396 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.012385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.021811 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.021781    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w59m6\""
Oct 30 22:00:42.023960 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.023931    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"]
Oct 30 22:00:42.113083 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113049    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szf88\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-kube-api-access-szf88\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113091    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113124    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113160    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113188    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538657b-f203-4b93-80d6-77522db77d56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113222    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113246    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113699 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113308    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538657b-f203-4b93-80d6-77522db77d56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113699 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113396    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113699 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113553    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113699 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113685    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113859 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113778    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113859 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113791    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.113980 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.113962    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538657b-f203-4b93-80d6-77522db77d56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.115576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.115552    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538657b-f203-4b93-80d6-77522db77d56-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.115831 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.115812    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538657b-f203-4b93-80d6-77522db77d56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.133243 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.133217    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.133416 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.133393    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szf88\" (UniqueName: \"kubernetes.io/projected/f538657b-f203-4b93-80d6-77522db77d56-kube-api-access-szf88\") pod \"maas-default-gateway-openshift-default-57f78bcd55-nkgpq\" (UID: \"f538657b-f203-4b93-80d6-77522db77d56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.308974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.308882    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:42.630748 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.630721    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"]
Oct 30 22:00:42.632385 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:00:42.632358    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf538657b_f203_4b93_80d6_77522db77d56.slice/crio-5f61a474d59480db3c495838aec70048ab48bdba724b3d737dac895c756a193f WatchSource:0}: Error finding container 5f61a474d59480db3c495838aec70048ab48bdba724b3d737dac895c756a193f: Status 404 returned error can't find the container with id 5f61a474d59480db3c495838aec70048ab48bdba724b3d737dac895c756a193f
Oct 30 22:00:42.635263 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.635211    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:00:42.635390 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.635312    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:00:42.635390 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.635357    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:00:42.730372 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.730332    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" event={"ID":"f538657b-f203-4b93-80d6-77522db77d56","Type":"ContainerStarted","Data":"ae29638811db3a274dbf9fe712ac72cc517665cc61826493905657414f5372b8"}
Oct 30 22:00:42.730597 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.730401    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" event={"ID":"f538657b-f203-4b93-80d6-77522db77d56","Type":"ContainerStarted","Data":"5f61a474d59480db3c495838aec70048ab48bdba724b3d737dac895c756a193f"}
Oct 30 22:00:42.773643 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.773587    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podStartSLOduration=1.7735724240000001 podStartE2EDuration="1.773572424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:42.7727032 +0000 UTC m=+3586.184242637" watchObservedRunningTime="2026-10-30 22:00:42.773572424 +0000 UTC m=+3586.185111861"
Oct 30 22:00:42.877143 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.877103    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"]
Oct 30 22:00:42.881038 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.880984    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:42.899017 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.898988    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:42.899175 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.899152    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:42.899237 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.899192    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:00:42.901619 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.901595    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:42.903759 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.903736    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"]
Oct 30 22:00:42.927983 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.927953    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7d8f\" (UniqueName: \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:42.928142 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.928010    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:42.928142 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:42.928136    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.031956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.028767    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.031956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.028866    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.031956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.028972    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7d8f\" (UniqueName: \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.031956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.029672    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.032679 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.032315    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.049225 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.049182    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7d8f\" (UniqueName: \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\") pod \"maas-api-revoke-sub-2e1f5b5f-qx5mf\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.194091 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.193998    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:43.310865 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.310835    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:43.312113 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.312094    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:43.312189 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.312140    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:43.515927 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.515898    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"]
Oct 30 22:00:43.517873 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:00:43.517844    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5bfb5a_590b_49e9_928f_ed79793d5991.slice/crio-d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce WatchSource:0}: Error finding container d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce: Status 404 returned error can't find the container with id d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce
Oct 30 22:00:43.754525 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:43.754473    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf" event={"ID":"3b5bfb5a-590b-49e9-928f-ed79793d5991","Type":"ContainerStarted","Data":"d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce"}
Oct 30 22:00:44.311291 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:44.311210    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:44.311291 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:44.311263    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:44.760243 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:44.760204    7037 generic.go:356] "Generic (PLEG): container finished" podID="3b5bfb5a-590b-49e9-928f-ed79793d5991" containerID="e0b3ed2465e36bda913b26232dc2beb83df3500796032a62f3dbc7febebf0b60" exitCode=0
Oct 30 22:00:44.760431 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:44.760258    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf" event={"ID":"3b5bfb5a-590b-49e9-928f-ed79793d5991","Type":"ContainerDied","Data":"e0b3ed2465e36bda913b26232dc2beb83df3500796032a62f3dbc7febebf0b60"}
Oct 30 22:00:45.311002 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.310967    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:45.311177 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.311022    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:45.914112 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.914089    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:45.969540 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.969509    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\" (UniqueName: \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\") pod \"3b5bfb5a-590b-49e9-928f-ed79793d5991\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") "
Oct 30 22:00:45.969825 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.969805    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\") pod \"3b5bfb5a-590b-49e9-928f-ed79793d5991\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") "
Oct 30 22:00:45.969984 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.969949    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\") pod \"3b5bfb5a-590b-49e9-928f-ed79793d5991\" (UID: \"3b5bfb5a-590b-49e9-928f-ed79793d5991\") "
Oct 30 22:00:45.970453 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.970119    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca" pod "3b5bfb5a-590b-49e9-928f-ed79793d5991" (UID: "3b5bfb5a-590b-49e9-928f-ed79793d5991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:45.970909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.970888    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:45.973517 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.971516    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f" pod "3b5bfb5a-590b-49e9-928f-ed79793d5991" (UID: "3b5bfb5a-590b-49e9-928f-ed79793d5991"). InnerVolumeSpecName "kube-api-access-w7d8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:45.973517 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:45.972076    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca" pod "3b5bfb5a-590b-49e9-928f-ed79793d5991" (UID: "3b5bfb5a-590b-49e9-928f-ed79793d5991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:46.071955 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.071921    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7d8f\" (UniqueName: \"kubernetes.io/projected/3b5bfb5a-590b-49e9-928f-ed79793d5991-kube-api-access-w7d8f\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:46.071955 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.071955    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5bfb5a-590b-49e9-928f-ed79793d5991-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:46.311036 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.310999    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:46.311225 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.311057    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:46.770857 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.770825    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"
Oct 30 22:00:46.771062 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.770859    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf" event={"ID":"3b5bfb5a-590b-49e9-928f-ed79793d5991","Type":"ContainerDied","Data":"d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce"}
Oct 30 22:00:46.771062 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:46.770913    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c77792583ecf26cd982d59ac9f64f35734e6196ce2d56a16e36c7cebe6f3ce"
Oct 30 22:00:47.310884 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:47.310845    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:47.311277 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:47.310909    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:48.311033 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:48.310996    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:48.311440 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:48.311048    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:49.311078 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:49.311039    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-nkgpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 22:00:49.311523 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:49.311099    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq" podUID="f538657b-f203-4b93-80d6-77522db77d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 22:00:50.840012 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:50.839981    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:50.840536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:50.840404    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:50.842168 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:50.842138    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-nkgpq"
Oct 30 22:00:53.971776 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.971734    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"]
Oct 30 22:00:53.972272 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.972231    7037 cpu_manager.go:414] "containerMap: removing container" podUID="3b5bfb5a-590b-49e9-928f-ed79793d5991" containerName="revoke-keys"
Oct 30 22:00:53.972272 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.972247    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5bfb5a-590b-49e9-928f-ed79793d5991" containerName="revoke-keys"
Oct 30 22:00:53.972351 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.972333    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5bfb5a-590b-49e9-928f-ed79793d5991" containerName="revoke-keys"
Oct 30 22:00:53.976862 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.976838    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:53.993526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:53.993464    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"]
Oct 30 22:00:54.004160 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.004128    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:54.004160 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.004155    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:00:54.034560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.034526    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:54.034742 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.034654    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:54.046112 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.046077    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6nc\" (UniqueName: \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.046281 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.046128    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.046281 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.046233    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.146875 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.146833    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6nc\" (UniqueName: \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.147075 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.146902    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.147075 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.146958    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.147634 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.147609    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.149340 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.149319    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.166699 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.166664    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6nc\" (UniqueName: \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\") pod \"maas-api-revoke-sub-ec843018-qcmsh\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.290780 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.290701    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:54.586441 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.586412    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"]
Oct 30 22:00:54.587025 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:00:54.586998    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652b9b5_4e66_4e9e_915b_6f2da40b98d5.slice/crio-2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96 WatchSource:0}: Error finding container 2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96: Status 404 returned error can't find the container with id 2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96
Oct 30 22:00:54.824908 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:54.824872    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh" event={"ID":"6652b9b5-4e66-4e9e-915b-6f2da40b98d5","Type":"ContainerStarted","Data":"2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96"}
Oct 30 22:00:55.832302 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:55.832267    7037 generic.go:356] "Generic (PLEG): container finished" podID="6652b9b5-4e66-4e9e-915b-6f2da40b98d5" containerID="ca0bc3aa58f63c0235d184f44c6a9cef4ecc93cd1d0a0f6ddcf4ca03ae25baf8" exitCode=0
Oct 30 22:00:55.832746 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:55.832353    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh" event={"ID":"6652b9b5-4e66-4e9e-915b-6f2da40b98d5","Type":"ContainerDied","Data":"ca0bc3aa58f63c0235d184f44c6a9cef4ecc93cd1d0a0f6ddcf4ca03ae25baf8"}
Oct 30 22:00:56.994381 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:56.994354    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:00:57.077400 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.077368    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\" (UniqueName: \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\") pod \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") "
Oct 30 22:00:57.077602 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.077428    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\") pod \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") "
Oct 30 22:00:57.077602 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.077462    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\") pod \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\" (UID: \"6652b9b5-4e66-4e9e-915b-6f2da40b98d5\") "
Oct 30 22:00:57.077907 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.077869    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca" pod "6652b9b5-4e66-4e9e-915b-6f2da40b98d5" (UID: "6652b9b5-4e66-4e9e-915b-6f2da40b98d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:57.079900 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.079876    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca" pod "6652b9b5-4e66-4e9e-915b-6f2da40b98d5" (UID: "6652b9b5-4e66-4e9e-915b-6f2da40b98d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:57.080224 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.080196    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc" pod "6652b9b5-4e66-4e9e-915b-6f2da40b98d5" (UID: "6652b9b5-4e66-4e9e-915b-6f2da40b98d5"). InnerVolumeSpecName "kube-api-access-5k6nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:57.178576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.178535    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:57.178576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.178566    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:57.178576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.178575    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6nc\" (UniqueName: \"kubernetes.io/projected/6652b9b5-4e66-4e9e-915b-6f2da40b98d5-kube-api-access-5k6nc\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:00:57.847088 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.847050    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh" event={"ID":"6652b9b5-4e66-4e9e-915b-6f2da40b98d5","Type":"ContainerDied","Data":"2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96"}
Oct 30 22:00:57.847380 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.847361    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a117cbddf5fffcd8609a1e965e7dee92200a5994d334804ea4819dbddf22e96"
Oct 30 22:00:57.847532 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:00:57.847089    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"
Oct 30 22:01:03.671213 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:01:03.671172    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:01:12.081454 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.081351    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"]
Oct 30 22:01:12.082125 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.082099    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6652b9b5-4e66-4e9e-915b-6f2da40b98d5" containerName="revoke-keys"
Oct 30 22:01:12.082236 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.082127    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6652b9b5-4e66-4e9e-915b-6f2da40b98d5" containerName="revoke-keys"
Oct 30 22:01:12.082307 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.082265    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6652b9b5-4e66-4e9e-915b-6f2da40b98d5" containerName="revoke-keys"
Oct 30 22:01:12.087234 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.087210    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.118638 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.118602    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"]
Oct 30 22:01:12.122269 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.122243    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"]
Oct 30 22:01:12.125909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.125882    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.148481 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.148440    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"]
Oct 30 22:01:12.226200 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226159    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.226411 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226232    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226411 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226284    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be548911-e631-4dac-85aa-591341df73f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226411 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226388    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226604 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226425    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.226604 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226452    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226604 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226582    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be548911-e631-4dac-85aa-591341df73f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226750 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226628    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.226795 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226740    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.226795 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226786    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.226891 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226823    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.226969 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.226947    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.227027 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227009    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.227087 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227068    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.227149 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227130    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb2r\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-kube-api-access-9nb2r\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.227204 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227163    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5jgl\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-kube-api-access-q5jgl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.227263 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227247    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.227308 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.227281    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328379 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328348    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328379 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328384    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be548911-e631-4dac-85aa-591341df73f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328420    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328446    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328472    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328615    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be548911-e631-4dac-85aa-591341df73f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328662    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328710    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328748    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328822    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328843    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328907    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.328938 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328919    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328966    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.328981    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329030    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329047    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329057    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329063    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb2r\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-kube-api-access-9nb2r\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329101    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5jgl\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-kube-api-access-q5jgl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329138    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329148    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329164    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.329260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329220    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.329897 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329403    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be548911-e631-4dac-85aa-591341df73f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.329897 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.329667    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.330130 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.330101    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.330305 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.330281    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.331299 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.331278    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be548911-e631-4dac-85aa-591341df73f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.331392 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.331370    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.331616 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.331567    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be548911-e631-4dac-85aa-591341df73f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.331832 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.331815    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.348594 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.348567    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.348921 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.348892    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb2r\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-kube-api-access-9nb2r\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.349081 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.349064    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be548911-e631-4dac-85aa-591341df73f8-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xgns5\" (UID: \"be548911-e631-4dac-85aa-591341df73f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.349155 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.349080    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5jgl\" (UniqueName: \"kubernetes.io/projected/c9eea0c1-4200-4348-bdf4-5df5c5bab262-kube-api-access-q5jgl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bz96z\" (UID: \"c9eea0c1-4200-4348-bdf4-5df5c5bab262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.402234 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.402196    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:12.451749 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.451702    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:12.708542 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.708513    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"]
Oct 30 22:01:12.709884 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:01:12.709860    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe548911_e631_4dac_85aa_591341df73f8.slice/crio-9ffc6ed051990e8b6957944dddf1316737c6d8a74c6c105e7c69055c21ff5be2 WatchSource:0}: Error finding container 9ffc6ed051990e8b6957944dddf1316737c6d8a74c6c105e7c69055c21ff5be2: Status 404 returned error can't find the container with id 9ffc6ed051990e8b6957944dddf1316737c6d8a74c6c105e7c69055c21ff5be2
Oct 30 22:01:12.712055 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.712020    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.712188 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.712093    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.712188 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.712130    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.755910 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.755882    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"]
Oct 30 22:01:12.756020 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:01:12.755929    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eea0c1_4200_4348_bdf4_5df5c5bab262.slice/crio-35199e5d147343a77927277aadf68b455418435396c80893eb0af6e467c44981 WatchSource:0}: Error finding container 35199e5d147343a77927277aadf68b455418435396c80893eb0af6e467c44981: Status 404 returned error can't find the container with id 35199e5d147343a77927277aadf68b455418435396c80893eb0af6e467c44981
Oct 30 22:01:12.758261 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.758229    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.758357 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.758293    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.758357 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.758321    7037 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:01:12.925772 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.925635    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" event={"ID":"be548911-e631-4dac-85aa-591341df73f8","Type":"ContainerStarted","Data":"afc9b245035f5059c2f2d52503902df6b847687924b3fb7e6d7f228c05957cd9"}
Oct 30 22:01:12.925772 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.925718    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" event={"ID":"be548911-e631-4dac-85aa-591341df73f8","Type":"ContainerStarted","Data":"9ffc6ed051990e8b6957944dddf1316737c6d8a74c6c105e7c69055c21ff5be2"}
Oct 30 22:01:12.927579 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.927553    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" event={"ID":"c9eea0c1-4200-4348-bdf4-5df5c5bab262","Type":"ContainerStarted","Data":"047a0eaad42db3c8b598d681480cdc02f592b6fc0aef8fe652c7dfdf14bb2dcd"}
Oct 30 22:01:12.927721 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.927593    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" event={"ID":"c9eea0c1-4200-4348-bdf4-5df5c5bab262","Type":"ContainerStarted","Data":"35199e5d147343a77927277aadf68b455418435396c80893eb0af6e467c44981"}
Oct 30 22:01:12.969837 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:12.969716    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podStartSLOduration=0.969697875 podStartE2EDuration="969.697875ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:12.969371348 +0000 UTC m=+3616.380910784" watchObservedRunningTime="2026-10-30 22:01:12.969697875 +0000 UTC m=+3616.381237315"
Oct 30 22:01:13.012690 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.012632    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podStartSLOduration=1.0126176120000001 podStartE2EDuration="1.012617612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:13.010387298 +0000 UTC m=+3616.421926735" watchObservedRunningTime="2026-10-30 22:01:13.012617612 +0000 UTC m=+3616.424157093"
Oct 30 22:01:13.403127 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.403082    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:13.404576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.404551    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:13.404649 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.404626    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:13.456515 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.456096    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:13.456515 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.456236    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:13.456515 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:13.456279    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:14.403347 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:14.403310    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:14.403747 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:14.403366    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:14.455940 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:14.455900    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:14.456248 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:14.456217    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:15.403236 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:15.403198    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:15.403428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:15.403257    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:15.453926 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:15.453885    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:15.454205 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:15.454174    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:16.403592 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:16.403548    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:16.404201 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:16.404164    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:16.453965 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:16.453922    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:16.454250 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:16.454220    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:17.411679 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:17.409568    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:17.411679 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:17.409642    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:17.474523 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:17.472630    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:17.474523 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:17.472696    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:18.402755 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:18.402716    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:18.402949 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:18.402771    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:18.457991 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:18.457950    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:18.458665 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:18.458618    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:19.403135 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:19.403094    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:19.403303 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:19.403164    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:19.452033 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:19.451997    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:19.452226 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:19.452049    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:20.403276 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:20.403237    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Oct 30 22:01:20.403734 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:20.403293    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Oct 30 22:01:20.452997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:20.452947    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Oct 30 22:01:20.453188 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:20.453025    7037 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Oct 30 22:01:21.407147 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.407110    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:21.407677 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.407509    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:21.408218 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.408197    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5"
Oct 30 22:01:21.455766 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.455738    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:21.456059 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.456032    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:21.456630 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:21.456606    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z"
Oct 30 22:01:23.344667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.344634    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"]
Oct 30 22:01:23.348798 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.348776    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.378378 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.378350    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:01:23.378378 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.378354    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:23.378625 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.378406    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:01:23.378625 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.378426    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:01:23.380247 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.380222    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"]
Oct 30 22:01:23.395395 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.395369    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.395552 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.395413    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.395552 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.395507    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhxf\" (UniqueName: \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.496664 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.496623    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.496832 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.496684    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.496832 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.496717    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhxf\" (UniqueName: \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.497291 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.497268    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.499221 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.499196    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.516830 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.516799    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhxf\" (UniqueName: \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\") pod \"maas-api-revoke-sub-046e1343-ktgzj\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.662237 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.662154    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:23.956856 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:01:23.956821    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adb619d_d686_419b_9811_3391e03152c5.slice/crio-8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b WatchSource:0}: Error finding container 8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b: Status 404 returned error can't find the container with id 8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b
Oct 30 22:01:23.957196 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:23.957167    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"]
Oct 30 22:01:24.029969 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:24.029918    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj" event={"ID":"2adb619d-d686-419b-9811-3391e03152c5","Type":"ContainerStarted","Data":"8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b"}
Oct 30 22:01:25.036581 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:25.036465    7037 generic.go:356] "Generic (PLEG): container finished" podID="2adb619d-d686-419b-9811-3391e03152c5" containerID="81a989b3637ec26cc115ee0f1ca510a065092802ee4dac458b6d8b1e66d89219" exitCode=0
Oct 30 22:01:25.036581 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:25.036547    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj" event={"ID":"2adb619d-d686-419b-9811-3391e03152c5","Type":"ContainerDied","Data":"81a989b3637ec26cc115ee0f1ca510a065092802ee4dac458b6d8b1e66d89219"}
Oct 30 22:01:26.171966 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.171939    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:26.225352 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.225324    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\") pod \"2adb619d-d686-419b-9811-3391e03152c5\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") "
Oct 30 22:01:26.225550 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.225395    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\") pod \"2adb619d-d686-419b-9811-3391e03152c5\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") "
Oct 30 22:01:26.225550 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.225463    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\" (UniqueName: \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\") pod \"2adb619d-d686-419b-9811-3391e03152c5\" (UID: \"2adb619d-d686-419b-9811-3391e03152c5\") "
Oct 30 22:01:26.225886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.225851    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca" pod "2adb619d-d686-419b-9811-3391e03152c5" (UID: "2adb619d-d686-419b-9811-3391e03152c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:26.227555 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.227528    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca" pod "2adb619d-d686-419b-9811-3391e03152c5" (UID: "2adb619d-d686-419b-9811-3391e03152c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:26.227660 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.227599    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf" pod "2adb619d-d686-419b-9811-3391e03152c5" (UID: "2adb619d-d686-419b-9811-3391e03152c5"). InnerVolumeSpecName "kube-api-access-2zhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:26.327267 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.327228    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhxf\" (UniqueName: \"kubernetes.io/projected/2adb619d-d686-419b-9811-3391e03152c5-kube-api-access-2zhxf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:26.327267 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.327264    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2adb619d-d686-419b-9811-3391e03152c5-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:26.327475 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:26.327280    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2adb619d-d686-419b-9811-3391e03152c5-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:27.055140 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:27.055049    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"
Oct 30 22:01:27.055140 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:27.055062    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj" event={"ID":"2adb619d-d686-419b-9811-3391e03152c5","Type":"ContainerDied","Data":"8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b"}
Oct 30 22:01:27.055343 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:27.055141    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8398075767cd4a38627ab03d8773630092d092919c2781138db1bfa7e815a52b"
Oct 30 22:01:33.673474 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.673436    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"]
Oct 30 22:01:33.674111 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.674089    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2adb619d-d686-419b-9811-3391e03152c5" containerName="revoke-keys"
Oct 30 22:01:33.674199 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.674113    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adb619d-d686-419b-9811-3391e03152c5" containerName="revoke-keys"
Oct 30 22:01:33.674255 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.674231    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adb619d-d686-419b-9811-3391e03152c5" containerName="revoke-keys"
Oct 30 22:01:33.676314 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:01:33.676283    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:01:33.677475 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.677454    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.697771 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.697739    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"]
Oct 30 22:01:33.707294 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.707258    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:01:33.707518 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.707299    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:33.707518 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.707362    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:01:33.707518 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.707418    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:01:33.804512 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.804444    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfb2\" (UniqueName: \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.804512 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.804514    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.804740 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.804633    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.905453 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.905417    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfb2\" (UniqueName: \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.905453 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.905458    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.905722 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.905549    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.906201 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.906175    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.907988 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.907969    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.926609 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.926542    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfb2\" (UniqueName: \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\") pod \"maas-api-revoke-sub-d3fba056-jckz7\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:33.990594 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:33.990557    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:34.288080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:34.288048    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"]
Oct 30 22:01:34.288626 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:01:34.288596    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00525b43_b221_421c_8027_a1ef120f1c45.slice/crio-9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea WatchSource:0}: Error finding container 9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea: Status 404 returned error can't find the container with id 9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea
Oct 30 22:01:35.100964 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:35.100926    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7" event={"ID":"00525b43-b221-421c-8027-a1ef120f1c45","Type":"ContainerStarted","Data":"8dd7c8d22dd51a8a28c11f91810a32ccc933e8d010a0f1556b29cc8e45e9d7bd"}
Oct 30 22:01:35.101400 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:35.100996    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7" event={"ID":"00525b43-b221-421c-8027-a1ef120f1c45","Type":"ContainerStarted","Data":"9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea"}
Oct 30 22:01:36.110868 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.110831    7037 generic.go:356] "Generic (PLEG): container finished" podID="00525b43-b221-421c-8027-a1ef120f1c45" containerID="8dd7c8d22dd51a8a28c11f91810a32ccc933e8d010a0f1556b29cc8e45e9d7bd" exitCode=0
Oct 30 22:01:36.111280 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.110879    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7" event={"ID":"00525b43-b221-421c-8027-a1ef120f1c45","Type":"ContainerDied","Data":"8dd7c8d22dd51a8a28c11f91810a32ccc933e8d010a0f1556b29cc8e45e9d7bd"}
Oct 30 22:01:36.441526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.431893    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:36.556167 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.556131    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\") pod \"00525b43-b221-421c-8027-a1ef120f1c45\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") "
Oct 30 22:01:36.556572 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.556546    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\" (UniqueName: \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\") pod \"00525b43-b221-421c-8027-a1ef120f1c45\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") "
Oct 30 22:01:36.556825 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.556806    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\") pod \"00525b43-b221-421c-8027-a1ef120f1c45\" (UID: \"00525b43-b221-421c-8027-a1ef120f1c45\") "
Oct 30 22:01:36.581105 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.581072    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca" pod "00525b43-b221-421c-8027-a1ef120f1c45" (UID: "00525b43-b221-421c-8027-a1ef120f1c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:36.594888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.594158    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca" pod "00525b43-b221-421c-8027-a1ef120f1c45" (UID: "00525b43-b221-421c-8027-a1ef120f1c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:36.594888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.594308    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2" pod "00525b43-b221-421c-8027-a1ef120f1c45" (UID: "00525b43-b221-421c-8027-a1ef120f1c45"). InnerVolumeSpecName "kube-api-access-7hfb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:36.662996 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.662818    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:36.662996 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.662863    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfb2\" (UniqueName: \"kubernetes.io/projected/00525b43-b221-421c-8027-a1ef120f1c45-kube-api-access-7hfb2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:36.662996 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:36.662880    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00525b43-b221-421c-8027-a1ef120f1c45-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:37.117080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.117046    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"
Oct 30 22:01:37.117080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.117075    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7" event={"ID":"00525b43-b221-421c-8027-a1ef120f1c45","Type":"ContainerDied","Data":"9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea"}
Oct 30 22:01:37.117617 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.117120    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c54803dd6f77df8abd38af63fe173cb3fe7792b4d42e92cd92263620ae86aea"
Oct 30 22:01:37.409026 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.408929    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:01:37.409026 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.408995    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:37.504707 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.504665    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:01:37.504899 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:37.504742    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:52.460516 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:52.458594    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:01:52.460516 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:52.458709    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:52.502794 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:52.502684    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:01:52.502794 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:52.502773    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:55.272083 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.272048    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"]
Oct 30 22:01:55.272585 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.272531    7037 cpu_manager.go:414] "containerMap: removing container" podUID="00525b43-b221-421c-8027-a1ef120f1c45" containerName="revoke-keys"
Oct 30 22:01:55.272585 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.272544    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00525b43-b221-421c-8027-a1ef120f1c45" containerName="revoke-keys"
Oct 30 22:01:55.272683 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.272650    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="00525b43-b221-421c-8027-a1ef120f1c45" containerName="revoke-keys"
Oct 30 22:01:55.278361 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.278339    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.294146 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.294118    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"]
Oct 30 22:01:55.304393 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.304363    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:55.304571 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.304408    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:01:55.304571 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.304411    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:01:55.305369 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.305349    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:01:55.330364 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.330328    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxvt\" (UniqueName: \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.330544 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.330407    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.330627 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.330606    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.431768 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.431727    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.431937 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.431868    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.431937 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.431897    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxvt\" (UniqueName: \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.432604 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.432575    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.434190 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.434171    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.451718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.451686    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxvt\" (UniqueName: \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\") pod \"maas-api-revoke-sub-ae594979-7sb46\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.593312 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.593277    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:55.887233 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:55.887197    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"]
Oct 30 22:01:55.887730 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:01:55.887704    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49c3063_65b3_4bfc_b174_846dc3b1532a.slice/crio-a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71 WatchSource:0}: Error finding container a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71: Status 404 returned error can't find the container with id a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71
Oct 30 22:01:56.231692 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:56.231609    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46" event={"ID":"b49c3063-65b3-4bfc-b174-846dc3b1532a","Type":"ContainerStarted","Data":"48b710236c04868cc714297f1ccec8419a0364e4b70c2d9ec57c6fbfc318a040"}
Oct 30 22:01:56.231692 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:56.231664    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46" event={"ID":"b49c3063-65b3-4bfc-b174-846dc3b1532a","Type":"ContainerStarted","Data":"a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71"}
Oct 30 22:01:56.270988 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:56.270937    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46" podStartSLOduration=1.270923477 podStartE2EDuration="1.270923477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:56.269781989 +0000 UTC m=+3659.681321493" watchObservedRunningTime="2026-10-30 22:01:56.270923477 +0000 UTC m=+3659.682462940"
Oct 30 22:01:57.238207 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:57.238171    7037 generic.go:356] "Generic (PLEG): container finished" podID="b49c3063-65b3-4bfc-b174-846dc3b1532a" containerID="48b710236c04868cc714297f1ccec8419a0364e4b70c2d9ec57c6fbfc318a040" exitCode=0
Oct 30 22:01:57.238657 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:57.238255    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46" event={"ID":"b49c3063-65b3-4bfc-b174-846dc3b1532a","Type":"ContainerDied","Data":"48b710236c04868cc714297f1ccec8419a0364e4b70c2d9ec57c6fbfc318a040"}
Oct 30 22:01:58.384283 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.384257    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:58.464859 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.464826    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\") pod \"b49c3063-65b3-4bfc-b174-846dc3b1532a\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") "
Oct 30 22:01:58.465041 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.464916    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\" (UniqueName: \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\") pod \"b49c3063-65b3-4bfc-b174-846dc3b1532a\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") "
Oct 30 22:01:58.465041 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.465016    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\") pod \"b49c3063-65b3-4bfc-b174-846dc3b1532a\" (UID: \"b49c3063-65b3-4bfc-b174-846dc3b1532a\") "
Oct 30 22:01:58.465411 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.465377    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca" pod "b49c3063-65b3-4bfc-b174-846dc3b1532a" (UID: "b49c3063-65b3-4bfc-b174-846dc3b1532a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:58.467029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.467004    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca" pod "b49c3063-65b3-4bfc-b174-846dc3b1532a" (UID: "b49c3063-65b3-4bfc-b174-846dc3b1532a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:58.467141 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.467003    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt" pod "b49c3063-65b3-4bfc-b174-846dc3b1532a" (UID: "b49c3063-65b3-4bfc-b174-846dc3b1532a"). InnerVolumeSpecName "kube-api-access-2cxvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:58.566029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.565953    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:58.566029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.565985    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c3063-65b3-4bfc-b174-846dc3b1532a-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:58.566029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:58.565995    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxvt\" (UniqueName: \"kubernetes.io/projected/b49c3063-65b3-4bfc-b174-846dc3b1532a-kube-api-access-2cxvt\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:01:59.249297 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:59.249254    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46" event={"ID":"b49c3063-65b3-4bfc-b174-846dc3b1532a","Type":"ContainerDied","Data":"a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71"}
Oct 30 22:01:59.249297 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:59.249280    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"
Oct 30 22:01:59.249550 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:01:59.249316    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a746c5226243fbb630c15efabad6256357ebefdc777bab5615586f8258b8eb71"
Oct 30 22:02:03.671778 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:02:03.671730    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:02:16.673909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.673873    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"]
Oct 30 22:02:16.674352 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.674338    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b49c3063-65b3-4bfc-b174-846dc3b1532a" containerName="revoke-keys"
Oct 30 22:02:16.674402 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.674355    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49c3063-65b3-4bfc-b174-846dc3b1532a" containerName="revoke-keys"
Oct 30 22:02:16.674440 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.674432    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49c3063-65b3-4bfc-b174-846dc3b1532a" containerName="revoke-keys"
Oct 30 22:02:16.677645 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.677619    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.696633 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.696606    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"]
Oct 30 22:02:16.706640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.706464    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:16.706640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.706476    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:16.706640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.706577    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:16.706886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.706826    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:02:16.758671 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.758629    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4g4\" (UniqueName: \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.758840 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.758700    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.758840 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.758812    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.860144 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.860108    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.860342 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.860212    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.860342 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.860279    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4g4\" (UniqueName: \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.860870 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.860849    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.862408 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.862382    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.880018 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.879993    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4g4\" (UniqueName: \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\") pod \"maas-api-revoke-sub-a4aef7d7-886gw\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:16.991130 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:16.991051    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:17.284554 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:02:17.284520    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b3ffdf_6d5b_4240_a3cd_faa9c4ca2ddc.slice/crio-5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b WatchSource:0}: Error finding container 5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b: Status 404 returned error can't find the container with id 5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b
Oct 30 22:02:17.284718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:17.284659    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"]
Oct 30 22:02:17.343762 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:17.343732    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw" event={"ID":"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc","Type":"ContainerStarted","Data":"5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b"}
Oct 30 22:02:18.349997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:18.349959    7037 generic.go:356] "Generic (PLEG): container finished" podID="28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" containerID="1a5a69de59352be0c3349cff6e932259cb97e07443b5aacb96e3b4b39a6c0559" exitCode=0
Oct 30 22:02:18.350413 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:18.350044    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw" event={"ID":"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc","Type":"ContainerDied","Data":"1a5a69de59352be0c3349cff6e932259cb97e07443b5aacb96e3b4b39a6c0559"}
Oct 30 22:02:19.504344 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.504319    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:19.589374 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.589341    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\" (UniqueName: \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\") pod \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") "
Oct 30 22:02:19.589549 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.589478    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\") pod \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") "
Oct 30 22:02:19.589685 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.589667    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\") pod \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\" (UID: \"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc\") "
Oct 30 22:02:19.590001 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.589976    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca" pod "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" (UID: "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:19.591539 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.591518    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca" pod "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" (UID: "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:19.591640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.591614    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4" pod "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" (UID: "28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc"). InnerVolumeSpecName "kube-api-access-zq4g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:19.690833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.690752    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:19.690833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.690782    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4g4\" (UniqueName: \"kubernetes.io/projected/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-kube-api-access-zq4g4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:19.690833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:19.690791    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:20.369804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:20.369765    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw" event={"ID":"28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc","Type":"ContainerDied","Data":"5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b"}
Oct 30 22:02:20.369804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:20.369807    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"
Oct 30 22:02:20.370033 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:20.369827    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e64717b5b0b7ec7a97a7afcc7d44b3c2c9d5d6bd9b3ae1e979f3ed8767ef81b"
Oct 30 22:02:33.672324 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:02:33.672282    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:02:37.872338 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.872257    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"]
Oct 30 22:02:37.872790 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.872773    7037 cpu_manager.go:414] "containerMap: removing container" podUID="28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" containerName="revoke-keys"
Oct 30 22:02:37.872836 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.872791    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" containerName="revoke-keys"
Oct 30 22:02:37.872873 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.872860    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" containerName="revoke-keys"
Oct 30 22:02:37.876064 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.876047    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:37.892963 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.892936    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"]
Oct 30 22:02:37.903250 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.903223    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:37.903250 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.903240    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:37.903250 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.903256    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:02:37.903478 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:37.903329    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:38.008300 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.008267    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.008515 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.008313    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.008515 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.008434    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcvf\" (UniqueName: \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.109897 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.109859    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcvf\" (UniqueName: \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.110079 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.109952    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.110079 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.109984    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.110645 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.110621    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.112337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.112309    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.129429 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.129369    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcvf\" (UniqueName: \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\") pod \"maas-api-revoke-sub-79805ca9-lbtr6\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.188750 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.188712    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:38.479552 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:38.479520    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"]
Oct 30 22:02:38.479672 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:02:38.479529    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74efb83_7240_4b19_9d41_a5d2138571d7.slice/crio-eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9 WatchSource:0}: Error finding container eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9: Status 404 returned error can't find the container with id eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9
Oct 30 22:02:39.483683 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:39.483643    7037 generic.go:356] "Generic (PLEG): container finished" podID="e74efb83-7240-4b19-9d41-a5d2138571d7" containerID="e58235692b6d64986ed2663aa50af95180d74330f05f9f3c15bab4638ea72353" exitCode=0
Oct 30 22:02:39.484122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:39.483682    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6" event={"ID":"e74efb83-7240-4b19-9d41-a5d2138571d7","Type":"ContainerDied","Data":"e58235692b6d64986ed2663aa50af95180d74330f05f9f3c15bab4638ea72353"}
Oct 30 22:02:39.484122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:39.483741    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6" event={"ID":"e74efb83-7240-4b19-9d41-a5d2138571d7","Type":"ContainerStarted","Data":"eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9"}
Oct 30 22:02:40.627817 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.627785    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:40.735337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.735299    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\") pod \"e74efb83-7240-4b19-9d41-a5d2138571d7\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") "
Oct 30 22:02:40.735576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.735419    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\" (UniqueName: \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\") pod \"e74efb83-7240-4b19-9d41-a5d2138571d7\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") "
Oct 30 22:02:40.735576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.735480    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\") pod \"e74efb83-7240-4b19-9d41-a5d2138571d7\" (UID: \"e74efb83-7240-4b19-9d41-a5d2138571d7\") "
Oct 30 22:02:40.735917 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.735888    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca" pod "e74efb83-7240-4b19-9d41-a5d2138571d7" (UID: "e74efb83-7240-4b19-9d41-a5d2138571d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:40.737576 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.737544    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca" pod "e74efb83-7240-4b19-9d41-a5d2138571d7" (UID: "e74efb83-7240-4b19-9d41-a5d2138571d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:40.737678 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.737643    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf" pod "e74efb83-7240-4b19-9d41-a5d2138571d7" (UID: "e74efb83-7240-4b19-9d41-a5d2138571d7"). InnerVolumeSpecName "kube-api-access-mrcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:40.836599 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.836567    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:40.836599 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.836596    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcvf\" (UniqueName: \"kubernetes.io/projected/e74efb83-7240-4b19-9d41-a5d2138571d7-kube-api-access-mrcvf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:40.836599 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:40.836607    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74efb83-7240-4b19-9d41-a5d2138571d7-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:41.494718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:41.494687    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"
Oct 30 22:02:41.494718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:41.494706    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6" event={"ID":"e74efb83-7240-4b19-9d41-a5d2138571d7","Type":"ContainerDied","Data":"eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9"}
Oct 30 22:02:41.494943 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:41.494753    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb7989f043e31d5e15a778e7d225f4eb4905a83f2003cacf74c1599c4e4c61e9"
Oct 30 22:02:52.454514 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:52.449956    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:52.454514 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:52.450025    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:52.505739 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:52.505685    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:52.505964 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:52.505755    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:54.719294 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.719257    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"]
Oct 30 22:02:54.719719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.719705    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e74efb83-7240-4b19-9d41-a5d2138571d7" containerName="revoke-keys"
Oct 30 22:02:54.719760 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.719720    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74efb83-7240-4b19-9d41-a5d2138571d7" containerName="revoke-keys"
Oct 30 22:02:54.719845 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.719835    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74efb83-7240-4b19-9d41-a5d2138571d7" containerName="revoke-keys"
Oct 30 22:02:54.723227 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.723208    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.751362 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.751327    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:02:54.752600 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.752564    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:54.753785 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.753759    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"]
Oct 30 22:02:54.790084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.790050    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:54.790084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.790050    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:54.815029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.815001    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t47v\" (UniqueName: \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.815222 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.815047    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.815222 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.815108    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.916703 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.916667    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t47v\" (UniqueName: \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.916930 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.916718    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.916930 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.916746    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.917434 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.917406    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.919350 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.919318    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:54.936099 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:54.936071    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t47v\" (UniqueName: \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\") pod \"maas-api-revoke-sub-0b7e8edb-59svf\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:55.037067 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:55.036973    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:55.330821 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:55.330765    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"]
Oct 30 22:02:55.614984 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:55.614893    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf" event={"ID":"0e80508b-898f-434d-808d-be1fe3ed5986","Type":"ContainerStarted","Data":"e7b7886c6a45fde1752ee104433622cc3cc56cb7d0453eda0f8a1a97e7b898b1"}
Oct 30 22:02:56.622219 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:56.622176    7037 generic.go:356] "Generic (PLEG): container finished" podID="0e80508b-898f-434d-808d-be1fe3ed5986" containerID="e9dd84ccd679038e9230a45a1a866ba805d673d6d02b7eefae83561ba454b186" exitCode=0
Oct 30 22:02:56.622734 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:56.622264    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf" event={"ID":"0e80508b-898f-434d-808d-be1fe3ed5986","Type":"ContainerDied","Data":"e9dd84ccd679038e9230a45a1a866ba805d673d6d02b7eefae83561ba454b186"}
Oct 30 22:02:57.758908 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.758885    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:57.850797 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.850765    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\") pod \"0e80508b-898f-434d-808d-be1fe3ed5986\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") "
Oct 30 22:02:57.851005 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.850837    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\") pod \"0e80508b-898f-434d-808d-be1fe3ed5986\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") "
Oct 30 22:02:57.851005 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.850998    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\" (UniqueName: \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\") pod \"0e80508b-898f-434d-808d-be1fe3ed5986\" (UID: \"0e80508b-898f-434d-808d-be1fe3ed5986\") "
Oct 30 22:02:57.851192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.851169    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca" pod "0e80508b-898f-434d-808d-be1fe3ed5986" (UID: "0e80508b-898f-434d-808d-be1fe3ed5986"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:57.851337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.851323    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:57.852902 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.852872    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca" pod "0e80508b-898f-434d-808d-be1fe3ed5986" (UID: "0e80508b-898f-434d-808d-be1fe3ed5986"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:57.853027 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.853006    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v" pod "0e80508b-898f-434d-808d-be1fe3ed5986" (UID: "0e80508b-898f-434d-808d-be1fe3ed5986"). InnerVolumeSpecName "kube-api-access-2t47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:57.952208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.952127    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t47v\" (UniqueName: \"kubernetes.io/projected/0e80508b-898f-434d-808d-be1fe3ed5986-kube-api-access-2t47v\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:57.952208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:57.952158    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e80508b-898f-434d-808d-be1fe3ed5986-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:02:58.634590 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:58.634557    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"
Oct 30 22:02:58.634816 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:58.634589    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf" event={"ID":"0e80508b-898f-434d-808d-be1fe3ed5986","Type":"ContainerDied","Data":"e7b7886c6a45fde1752ee104433622cc3cc56cb7d0453eda0f8a1a97e7b898b1"}
Oct 30 22:02:58.634816 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:02:58.634638    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7b7886c6a45fde1752ee104433622cc3cc56cb7d0453eda0f8a1a97e7b898b1"
Oct 30 22:03:03.673161 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:03:03.673123    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:03:05.544894 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.544860    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"]
Oct 30 22:03:05.545391 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.545312    7037 cpu_manager.go:414] "containerMap: removing container" podUID="0e80508b-898f-434d-808d-be1fe3ed5986" containerName="revoke-keys"
Oct 30 22:03:05.545391 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.545324    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e80508b-898f-434d-808d-be1fe3ed5986" containerName="revoke-keys"
Oct 30 22:03:05.545469 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.545393    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e80508b-898f-434d-808d-be1fe3ed5986" containerName="revoke-keys"
Oct 30 22:03:05.548273 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.548255    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.567915 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.567881    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"]
Oct 30 22:03:05.577465 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.577436    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:03:05.577465 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.577437    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:05.578428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.578408    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:05.578428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.578428    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:05.631587 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.631548    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.631774 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.631619    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2mf\" (UniqueName: \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.631774 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.631705    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.732833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.732790    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2mf\" (UniqueName: \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.732833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.732841    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.733072 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.732902    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.733547 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.733486    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.735321 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.735296    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.752909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.752876    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2mf\" (UniqueName: \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\") pod \"maas-api-revoke-sub-3d4156f1-qpmw8\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:05.861888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:05.861849    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:06.156125 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:06.156090    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"]
Oct 30 22:03:06.156634 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:03:06.156609    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03211036_989f_425a_8b18_a0edd4bdedb4.slice/crio-5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088 WatchSource:0}: Error finding container 5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088: Status 404 returned error can't find the container with id 5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088
Oct 30 22:03:06.677474 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:06.677436    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8" event={"ID":"03211036-989f-425a-8b18-a0edd4bdedb4","Type":"ContainerStarted","Data":"061cba0c72da798c635001a06a66d7b8c2aa69effc6f897cd1ca559280526f01"}
Oct 30 22:03:06.677975 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:06.677517    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8" event={"ID":"03211036-989f-425a-8b18-a0edd4bdedb4","Type":"ContainerStarted","Data":"5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088"}
Oct 30 22:03:06.717370 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:06.717269    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8" podStartSLOduration=1.7172571269999999 podStartE2EDuration="1.717257127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:06.716816615 +0000 UTC m=+3730.128356051" watchObservedRunningTime="2026-10-30 22:03:06.717257127 +0000 UTC m=+3730.128796561"
Oct 30 22:03:07.684117 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:07.684079    7037 generic.go:356] "Generic (PLEG): container finished" podID="03211036-989f-425a-8b18-a0edd4bdedb4" containerID="061cba0c72da798c635001a06a66d7b8c2aa69effc6f897cd1ca559280526f01" exitCode=0
Oct 30 22:03:07.684535 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:07.684162    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8" event={"ID":"03211036-989f-425a-8b18-a0edd4bdedb4","Type":"ContainerDied","Data":"061cba0c72da798c635001a06a66d7b8c2aa69effc6f897cd1ca559280526f01"}
Oct 30 22:03:08.829876 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.829848    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:08.968113 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.968020    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\") pod \"03211036-989f-425a-8b18-a0edd4bdedb4\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") "
Oct 30 22:03:08.968113 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.968088    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\") pod \"03211036-989f-425a-8b18-a0edd4bdedb4\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") "
Oct 30 22:03:08.968370 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.968196    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\" (UniqueName: \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\") pod \"03211036-989f-425a-8b18-a0edd4bdedb4\" (UID: \"03211036-989f-425a-8b18-a0edd4bdedb4\") "
Oct 30 22:03:08.968432 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.968412    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca" pod "03211036-989f-425a-8b18-a0edd4bdedb4" (UID: "03211036-989f-425a-8b18-a0edd4bdedb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:08.968648 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.968627    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:08.970273 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.970242    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca" pod "03211036-989f-425a-8b18-a0edd4bdedb4" (UID: "03211036-989f-425a-8b18-a0edd4bdedb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:08.970366 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:08.970251    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf" pod "03211036-989f-425a-8b18-a0edd4bdedb4" (UID: "03211036-989f-425a-8b18-a0edd4bdedb4"). InnerVolumeSpecName "kube-api-access-sx2mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:09.069804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:09.069761    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2mf\" (UniqueName: \"kubernetes.io/projected/03211036-989f-425a-8b18-a0edd4bdedb4-kube-api-access-sx2mf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:09.069804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:09.069797    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03211036-989f-425a-8b18-a0edd4bdedb4-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:09.696291 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:09.696259    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"
Oct 30 22:03:09.696524 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:09.696260    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8" event={"ID":"03211036-989f-425a-8b18-a0edd4bdedb4","Type":"ContainerDied","Data":"5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088"}
Oct 30 22:03:09.696524 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:09.696390    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5865c9811b96ffc15c02ceb117ffe6e20cc3dd09c768304112dcccd9eff5b088"
Oct 30 22:03:22.454517 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:22.453615    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xgns5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:22.454517 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:22.453694    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xgns5" podUID="be548911-e631-4dac-85aa-591341df73f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:22.503714 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:22.503658    7037 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bz96z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:22.504069 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:22.504037    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bz96z" podUID="c9eea0c1-4200-4348-bdf4-5df5c5bab262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:26.960048 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:26.960005    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"]
Oct 30 22:03:26.972286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:26.972251    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpmw8"]
Oct 30 22:03:27.004017 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.003986    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"]
Oct 30 22:03:27.004547 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.004532    7037 cpu_manager.go:414] "containerMap: removing container" podUID="03211036-989f-425a-8b18-a0edd4bdedb4" containerName="revoke-keys"
Oct 30 22:03:27.004612 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.004549    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03211036-989f-425a-8b18-a0edd4bdedb4" containerName="revoke-keys"
Oct 30 22:03:27.004654 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.004622    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="03211036-989f-425a-8b18-a0edd4bdedb4" containerName="revoke-keys"
Oct 30 22:03:27.007932 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.007913    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.025339 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.025300    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:27.025560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.025387    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:27.025560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.025428    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:03:27.025707 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.025582    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:27.026627 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.026605    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"]
Oct 30 22:03:27.146506 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.146459    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.146698 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.146623    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.146698 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.146690    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wz9\" (UniqueName: \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.247785 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.247694    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.247785 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.247763    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.247994 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.247792    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wz9\" (UniqueName: \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.248584 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.248560    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.250168 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.250151    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.268035 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.268006    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wz9\" (UniqueName: \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\") pod \"maas-api-revoke-sub-3d4156f1-7q4t2\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.323519 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.323470    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:27.619043 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:03:27.619007    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7779901_cdae_47f9_972d_253844aaa161.slice/crio-fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe WatchSource:0}: Error finding container fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe: Status 404 returned error can't find the container with id fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe
Oct 30 22:03:27.619380 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.619356    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"]
Oct 30 22:03:27.833193 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:27.833158    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2" event={"ID":"f7779901-cdae-47f9-972d-253844aaa161","Type":"ContainerStarted","Data":"fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe"}
Oct 30 22:03:28.706750 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:28.706663    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03211036-989f-425a-8b18-a0edd4bdedb4" path="/var/lib/kubelet/pods/03211036-989f-425a-8b18-a0edd4bdedb4/volumes"
Oct 30 22:03:28.841451 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:28.841409    7037 generic.go:356] "Generic (PLEG): container finished" podID="f7779901-cdae-47f9-972d-253844aaa161" containerID="fef83670fe2bae8cb9f81281d20712778b27e7fdaabc47298ac5ee9df9c2a079" exitCode=0
Oct 30 22:03:28.841667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:28.841516    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2" event={"ID":"f7779901-cdae-47f9-972d-253844aaa161","Type":"ContainerDied","Data":"fef83670fe2bae8cb9f81281d20712778b27e7fdaabc47298ac5ee9df9c2a079"}
Oct 30 22:03:29.983922 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:29.983898    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:30.076573 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.076536    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\" (UniqueName: \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\") pod \"f7779901-cdae-47f9-972d-253844aaa161\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") "
Oct 30 22:03:30.076755 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.076725    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\") pod \"f7779901-cdae-47f9-972d-253844aaa161\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") "
Oct 30 22:03:30.076835 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.076818    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\") pod \"f7779901-cdae-47f9-972d-253844aaa161\" (UID: \"f7779901-cdae-47f9-972d-253844aaa161\") "
Oct 30 22:03:30.077122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.077093    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca" pod "f7779901-cdae-47f9-972d-253844aaa161" (UID: "f7779901-cdae-47f9-972d-253844aaa161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:30.078784 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.078751    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9" pod "f7779901-cdae-47f9-972d-253844aaa161" (UID: "f7779901-cdae-47f9-972d-253844aaa161"). InnerVolumeSpecName "kube-api-access-g4wz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:30.078916 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.078901    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca" pod "f7779901-cdae-47f9-972d-253844aaa161" (UID: "f7779901-cdae-47f9-972d-253844aaa161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:30.178075 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.177981    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7779901-cdae-47f9-972d-253844aaa161-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:30.178075 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.178015    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4wz9\" (UniqueName: \"kubernetes.io/projected/f7779901-cdae-47f9-972d-253844aaa161-kube-api-access-g4wz9\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:30.178075 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.178024    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7779901-cdae-47f9-972d-253844aaa161-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:30.852856 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.852826    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"
Oct 30 22:03:30.853038 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.852854    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2" event={"ID":"f7779901-cdae-47f9-972d-253844aaa161","Type":"ContainerDied","Data":"fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe"}
Oct 30 22:03:30.853038 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:30.852904    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe9bd099c340a92654d4dbe5e23b119441222116baa1c9fea1642bc548d505fe"
Oct 30 22:03:33.671332 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:03:33.671293    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:03:41.036213 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:41.036176    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"]
Oct 30 22:03:41.048505 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:41.048458    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab6bf0d5-4j4rm"]
Oct 30 22:03:42.706254 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:42.706219    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ebe198-0946-49cf-bd24-5be387389455" path="/var/lib/kubelet/pods/83ebe198-0946-49cf-bd24-5be387389455/volumes"
Oct 30 22:03:54.992449 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:54.992412    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"]
Oct 30 22:03:54.992997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:54.992892    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f7779901-cdae-47f9-972d-253844aaa161" containerName="revoke-keys"
Oct 30 22:03:54.992997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:54.992905    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7779901-cdae-47f9-972d-253844aaa161" containerName="revoke-keys"
Oct 30 22:03:54.993080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:54.993001    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7779901-cdae-47f9-972d-253844aaa161" containerName="revoke-keys"
Oct 30 22:03:54.996630 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:54.996612    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.014874 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.014846    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"]
Oct 30 22:03:55.024610 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.024580    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:55.024610 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.024588    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:55.024814 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.024637    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:55.024814 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.024688    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:03:55.046377 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.046335    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfhr\" (UniqueName: \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.046575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.046448    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.046575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.046540    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.147360 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.147319    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfhr\" (UniqueName: \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.147597 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.147380    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.147597 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.147408    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.148044 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.148019    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.149862 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.149840    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.167328 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.167300    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfhr\" (UniqueName: \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\") pod \"maas-api-revoke-sub-e2fc9c5b-75gn2\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.309807 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.309722    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:55.605481 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:55.605451    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"]
Oct 30 22:03:55.605692 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:03:55.605668    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19cd566_3169_4547_b70e_7306a9eb1aa5.slice/crio-b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b WatchSource:0}: Error finding container b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b: Status 404 returned error can't find the container with id b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b
Oct 30 22:03:56.024220 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:56.024178    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2" event={"ID":"f19cd566-3169-4547-b70e-7306a9eb1aa5","Type":"ContainerStarted","Data":"a4a4191ebc41400a102f44b9dcc66c81d63bd151d9f67763f5479f4966dbd299"}
Oct 30 22:03:56.024642 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:56.024243    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2" event={"ID":"f19cd566-3169-4547-b70e-7306a9eb1aa5","Type":"ContainerStarted","Data":"b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b"}
Oct 30 22:03:56.065534 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:56.065464    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2" podStartSLOduration=2.065451602 podStartE2EDuration="2.065451602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.063698007 +0000 UTC m=+3779.475237443" watchObservedRunningTime="2026-10-30 22:03:56.065451602 +0000 UTC m=+3779.476991040"
Oct 30 22:03:57.031353 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:57.031320    7037 generic.go:356] "Generic (PLEG): container finished" podID="f19cd566-3169-4547-b70e-7306a9eb1aa5" containerID="a4a4191ebc41400a102f44b9dcc66c81d63bd151d9f67763f5479f4966dbd299" exitCode=0
Oct 30 22:03:57.031781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:57.031402    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2" event={"ID":"f19cd566-3169-4547-b70e-7306a9eb1aa5","Type":"ContainerDied","Data":"a4a4191ebc41400a102f44b9dcc66c81d63bd151d9f67763f5479f4966dbd299"}
Oct 30 22:03:58.170339 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.170313    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:58.278025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.277987    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\" (UniqueName: \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\") pod \"f19cd566-3169-4547-b70e-7306a9eb1aa5\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") "
Oct 30 22:03:58.278235 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.278035    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\") pod \"f19cd566-3169-4547-b70e-7306a9eb1aa5\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") "
Oct 30 22:03:58.278235 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.278124    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\") pod \"f19cd566-3169-4547-b70e-7306a9eb1aa5\" (UID: \"f19cd566-3169-4547-b70e-7306a9eb1aa5\") "
Oct 30 22:03:58.278533 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.278480    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca" pod "f19cd566-3169-4547-b70e-7306a9eb1aa5" (UID: "f19cd566-3169-4547-b70e-7306a9eb1aa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:58.280264 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.280240    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca" pod "f19cd566-3169-4547-b70e-7306a9eb1aa5" (UID: "f19cd566-3169-4547-b70e-7306a9eb1aa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:58.280350 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.280329    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr" pod "f19cd566-3169-4547-b70e-7306a9eb1aa5" (UID: "f19cd566-3169-4547-b70e-7306a9eb1aa5"). InnerVolumeSpecName "kube-api-access-fvfhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:58.379593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.379559    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfhr\" (UniqueName: \"kubernetes.io/projected/f19cd566-3169-4547-b70e-7306a9eb1aa5-kube-api-access-fvfhr\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:58.379593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.379587    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:58.379593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:58.379597    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19cd566-3169-4547-b70e-7306a9eb1aa5-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.044116 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:59.044033    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"
Oct 30 22:03:59.044276 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:59.044032    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2" event={"ID":"f19cd566-3169-4547-b70e-7306a9eb1aa5","Type":"ContainerDied","Data":"b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b"}
Oct 30 22:03:59.044276 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:03:59.044165    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06198bf0130c56cbf1b8bdafd30a1d5751a9233c780c362ab94672b2011323b"
Oct 30 22:04:03.671317 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:04:03.671273    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:04:05.445999 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.445960    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"]
Oct 30 22:04:05.446554 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.446392    7037 cpu_manager.go:414] "containerMap: removing container" podUID="f19cd566-3169-4547-b70e-7306a9eb1aa5" containerName="revoke-keys"
Oct 30 22:04:05.446554 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.446403    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19cd566-3169-4547-b70e-7306a9eb1aa5" containerName="revoke-keys"
Oct 30 22:04:05.446554 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.446475    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19cd566-3169-4547-b70e-7306a9eb1aa5" containerName="revoke-keys"
Oct 30 22:04:05.449643 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.449620    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.469319 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.469290    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"]
Oct 30 22:04:05.479286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.479251    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:05.479286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.479274    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:05.479286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.479262    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:05.479582 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.479376    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:04:05.551455 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.551417    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rgm\" (UniqueName: \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.551672 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.551610    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.551672 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.551636    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.652874 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.652837    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.652874 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.652875    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.653107 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.652912    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rgm\" (UniqueName: \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.653562 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.653533    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.655258 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.655237    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.674442 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.674409    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rgm\" (UniqueName: \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\") pod \"maas-api-revoke-sub-5a29efb3-2g7lg\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:05.763688 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:05.763581    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:06.065268 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:06.065227    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"]
Oct 30 22:04:06.065759 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:04:06.065730    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16723ad6_5174_4368_8230_9bc885f227b3.slice/crio-c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c WatchSource:0}: Error finding container c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c: Status 404 returned error can't find the container with id c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c
Oct 30 22:04:06.084778 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:06.084731    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg" event={"ID":"16723ad6-5174-4368-8230-9bc885f227b3","Type":"ContainerStarted","Data":"c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c"}
Oct 30 22:04:07.093989 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:07.093957    7037 generic.go:356] "Generic (PLEG): container finished" podID="16723ad6-5174-4368-8230-9bc885f227b3" containerID="4d87851d0aa6d6bbdcdea246ee3350201e86d71c1af9c4768e2e156f074d5cd7" exitCode=0
Oct 30 22:04:07.094537 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:07.094065    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg" event={"ID":"16723ad6-5174-4368-8230-9bc885f227b3","Type":"ContainerDied","Data":"4d87851d0aa6d6bbdcdea246ee3350201e86d71c1af9c4768e2e156f074d5cd7"}
Oct 30 22:04:08.501420 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.501051    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:08.612258 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.612207    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\") pod \"16723ad6-5174-4368-8230-9bc885f227b3\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") "
Oct 30 22:04:08.612474 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.612305    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\") pod \"16723ad6-5174-4368-8230-9bc885f227b3\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") "
Oct 30 22:04:08.612474 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.612416    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\" (UniqueName: \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\") pod \"16723ad6-5174-4368-8230-9bc885f227b3\" (UID: \"16723ad6-5174-4368-8230-9bc885f227b3\") "
Oct 30 22:04:08.613764 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.613732    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca" pod "16723ad6-5174-4368-8230-9bc885f227b3" (UID: "16723ad6-5174-4368-8230-9bc885f227b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:08.614908 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.614880    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm" pod "16723ad6-5174-4368-8230-9bc885f227b3" (UID: "16723ad6-5174-4368-8230-9bc885f227b3"). InnerVolumeSpecName "kube-api-access-67rgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:08.615985 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.615956    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca" pod "16723ad6-5174-4368-8230-9bc885f227b3" (UID: "16723ad6-5174-4368-8230-9bc885f227b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:08.714219 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.714140    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rgm\" (UniqueName: \"kubernetes.io/projected/16723ad6-5174-4368-8230-9bc885f227b3-kube-api-access-67rgm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:08.714219 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.714170    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:08.714219 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:08.714180    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16723ad6-5174-4368-8230-9bc885f227b3-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:09.110328 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:09.110293    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"
Oct 30 22:04:09.110540 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:09.110327    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg" event={"ID":"16723ad6-5174-4368-8230-9bc885f227b3","Type":"ContainerDied","Data":"c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c"}
Oct 30 22:04:09.110540 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:09.110377    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a0ba331832580b2dcd1e185a3acc676bfec439fc88a9ca282372ce7b56492c"
Oct 30 22:04:24.654501 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.654453    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"]
Oct 30 22:04:24.666238 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.666202    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7q4t2"]
Oct 30 22:04:24.697759 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.697719    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"]
Oct 30 22:04:24.698216 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.698197    7037 cpu_manager.go:414] "containerMap: removing container" podUID="16723ad6-5174-4368-8230-9bc885f227b3" containerName="revoke-keys"
Oct 30 22:04:24.698216 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.698212    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16723ad6-5174-4368-8230-9bc885f227b3" containerName="revoke-keys"
Oct 30 22:04:24.698409 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.698283    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="16723ad6-5174-4368-8230-9bc885f227b3" containerName="revoke-keys"
Oct 30 22:04:24.701759 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.701732    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.709194 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.709164    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7779901-cdae-47f9-972d-253844aaa161" path="/var/lib/kubelet/pods/f7779901-cdae-47f9-972d-253844aaa161/volumes"
Oct 30 22:04:24.718817 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.718761    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:24.718817 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.718784    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:24.719044 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.719024    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:24.721299 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.721270    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"]
Oct 30 22:04:24.730581 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.730555    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:04:24.872251 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.872217    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.872454 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.872276    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mb9l\" (UniqueName: \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.872544 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.872523    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.973799 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.973712    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.973799 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.973763    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mb9l\" (UniqueName: \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.973996 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.973847    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.974548 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.974516    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.976215 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.976193    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:24.994974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:24.994939    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mb9l\" (UniqueName: \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\") pod \"maas-api-revoke-sub-3d4156f1-cmvn5\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:25.016772 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:25.016740    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:25.312809 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:04:25.312765    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca8cf20_c201_4605_9497_efbdf0614a7b.slice/crio-c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06 WatchSource:0}: Error finding container c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06: Status 404 returned error can't find the container with id c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06
Oct 30 22:04:25.313722 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:25.313693    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"]
Oct 30 22:04:26.202584 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:26.202539    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5" event={"ID":"5ca8cf20-c201-4605-9497-efbdf0614a7b","Type":"ContainerStarted","Data":"b99700050357ba89a209f9c1bd1e541b6e51af8f1605d9b6191988c3c4f135ba"}
Oct 30 22:04:26.203028 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:26.202610    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5" event={"ID":"5ca8cf20-c201-4605-9497-efbdf0614a7b","Type":"ContainerStarted","Data":"c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06"}
Oct 30 22:04:26.242793 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:26.242742    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5" podStartSLOduration=2.242728186 podStartE2EDuration="2.242728186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:26.24163409 +0000 UTC m=+3809.653173528" watchObservedRunningTime="2026-10-30 22:04:26.242728186 +0000 UTC m=+3809.654267623"
Oct 30 22:04:27.209811 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:27.209775    7037 generic.go:356] "Generic (PLEG): container finished" podID="5ca8cf20-c201-4605-9497-efbdf0614a7b" containerID="b99700050357ba89a209f9c1bd1e541b6e51af8f1605d9b6191988c3c4f135ba" exitCode=0
Oct 30 22:04:27.210235 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:27.209861    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5" event={"ID":"5ca8cf20-c201-4605-9497-efbdf0614a7b","Type":"ContainerDied","Data":"b99700050357ba89a209f9c1bd1e541b6e51af8f1605d9b6191988c3c4f135ba"}
Oct 30 22:04:28.348907 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.348878    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:28.411131 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.411086    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\") pod \"5ca8cf20-c201-4605-9497-efbdf0614a7b\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") "
Oct 30 22:04:28.411333 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.411191    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\") pod \"5ca8cf20-c201-4605-9497-efbdf0614a7b\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") "
Oct 30 22:04:28.411395 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.411333    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\" (UniqueName: \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\") pod \"5ca8cf20-c201-4605-9497-efbdf0614a7b\" (UID: \"5ca8cf20-c201-4605-9497-efbdf0614a7b\") "
Oct 30 22:04:28.411888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.411861    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca" pod "5ca8cf20-c201-4605-9497-efbdf0614a7b" (UID: "5ca8cf20-c201-4605-9497-efbdf0614a7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:28.414655 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.414624    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l" pod "5ca8cf20-c201-4605-9497-efbdf0614a7b" (UID: "5ca8cf20-c201-4605-9497-efbdf0614a7b"). InnerVolumeSpecName "kube-api-access-9mb9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:28.416822 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.416783    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca" pod "5ca8cf20-c201-4605-9497-efbdf0614a7b" (UID: "5ca8cf20-c201-4605-9497-efbdf0614a7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:28.512536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.512435    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mb9l\" (UniqueName: \"kubernetes.io/projected/5ca8cf20-c201-4605-9497-efbdf0614a7b-kube-api-access-9mb9l\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:28.512536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.512469    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:28.512536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:28.512481    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca8cf20-c201-4605-9497-efbdf0614a7b-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:04:29.222661 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:29.222621    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5" event={"ID":"5ca8cf20-c201-4605-9497-efbdf0614a7b","Type":"ContainerDied","Data":"c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06"}
Oct 30 22:04:29.222841 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:29.222671    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cf8d33abed47b59208cf1c4d193e17d2a3b822448a75a47d79e66cfa321f06"
Oct 30 22:04:29.222841 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:29.222632    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"
Oct 30 22:04:30.039307 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:30.039271    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"]
Oct 30 22:04:30.051927 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:30.051895    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qcbdm"]
Oct 30 22:04:30.706317 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:30.706285    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad08b54f-4278-4e8b-917c-56e64abeb0d2" path="/var/lib/kubelet/pods/ad08b54f-4278-4e8b-917c-56e64abeb0d2/volumes"
Oct 30 22:04:33.673047 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:04:33.673004    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:04:46.054045 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.054001    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:04:46.054686 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.054459    7037 cpu_manager.go:414] "containerMap: removing container" podUID="5ca8cf20-c201-4605-9497-efbdf0614a7b" containerName="revoke-keys"
Oct 30 22:04:46.054686 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.054472    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca8cf20-c201-4605-9497-efbdf0614a7b" containerName="revoke-keys"
Oct 30 22:04:46.054686 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.054590    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca8cf20-c201-4605-9497-efbdf0614a7b" containerName="revoke-keys"
Oct 30 22:04:46.058230 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.058208    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.075191 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.075159    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:46.087473 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.087441    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d36cd3-serving-cert\""
Oct 30 22:04:46.087473 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.087450    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pdl4\""
Oct 30 22:04:46.087706 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.087484    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:46.087706 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.087568    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:04:46.089691 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.089670    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:04:46.194554 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.194516    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.194761 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.194596    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptdz\" (UniqueName: \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.194761 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.194677    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.194761 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.194715    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.295983 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.295944    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptdz\" (UniqueName: \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.296199 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.296006    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.296199 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.296036    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.296199 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.296110    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.296770 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.296743    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.298601 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.298574    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.298758 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.298734    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.316998 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.316934    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptdz\" (UniqueName: \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\") pod \"maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.371793 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.371758    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:46.440343 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.440257    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:04:46.444816 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.444785    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.471525 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.468084    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-459ec8f2-dockercfg-gwtk2\""
Oct 30 22:04:46.471525 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.468422    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d36cd3\""
Oct 30 22:04:46.471525 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.469705    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:04:46.498132 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.498101    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99v5r\" (UniqueName: \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.498314 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.498180    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.599272 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.599224    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.599466 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.599383    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99v5r\" (UniqueName: \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.600165 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.600140    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.626077 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.626044    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99v5r\" (UniqueName: \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\") pod \"payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:46.683005 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.682973    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:04:46.683483 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:04:46.683460    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3765a30_a8a0_495e_823a_253dcebc53a5.slice/crio-e3502622a24ce5d561adec1c16047525a08077ce0efeb1da184be123fa90ca5a WatchSource:0}: Error finding container e3502622a24ce5d561adec1c16047525a08077ce0efeb1da184be123fa90ca5a: Status 404 returned error can't find the container with id e3502622a24ce5d561adec1c16047525a08077ce0efeb1da184be123fa90ca5a
Oct 30 22:04:46.684963 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.684939    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:04:46.762544 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:46.762510    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:47.135346 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.135312    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:04:47.137539 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:04:47.137509    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46720ecb_46bc_4a77_a218_860c7006d18d.slice/crio-cc273912fdcd86212e32b6a0934c31cc3d85a19aebaf552738c7b164c74aec33 WatchSource:0}: Error finding container cc273912fdcd86212e32b6a0934c31cc3d85a19aebaf552738c7b164c74aec33: Status 404 returned error can't find the container with id cc273912fdcd86212e32b6a0934c31cc3d85a19aebaf552738c7b164c74aec33
Oct 30 22:04:47.321477 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.321438    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" event={"ID":"e3765a30-a8a0-495e-823a-253dcebc53a5","Type":"ContainerStarted","Data":"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"}
Oct 30 22:04:47.321703 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.321511    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" event={"ID":"e3765a30-a8a0-495e-823a-253dcebc53a5","Type":"ContainerStarted","Data":"e3502622a24ce5d561adec1c16047525a08077ce0efeb1da184be123fa90ca5a"}
Oct 30 22:04:47.321703 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.321636    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:47.323146 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.323114    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" event={"ID":"46720ecb-46bc-4a77-a218-860c7006d18d","Type":"ContainerStarted","Data":"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"}
Oct 30 22:04:47.323277 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.323170    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:47.323277 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.323194    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" event={"ID":"46720ecb-46bc-4a77-a218-860c7006d18d","Type":"ContainerStarted","Data":"cc273912fdcd86212e32b6a0934c31cc3d85a19aebaf552738c7b164c74aec33"}
Oct 30 22:04:47.378242 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.378192    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" podStartSLOduration=0.890754913 podStartE2EDuration="1.378176528s" totalImagesPullingTime="487.421615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:47.377561984 +0000 UTC m=+3830.789101420" watchObservedRunningTime="2026-10-30 22:04:47.378176528 +0000 UTC m=+3830.789715965"
Oct 30 22:04:47.436457 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:47.436361    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" podStartSLOduration=1.4363472050000001 podStartE2EDuration="1.436347205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:47.434852219 +0000 UTC m=+3830.846391660" watchObservedRunningTime="2026-10-30 22:04:47.436347205 +0000 UTC m=+3830.847886636"
Oct 30 22:04:53.335208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:53.335178    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:04:58.333844 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:58.333812    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:04:59.378935 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:59.378897    7037 scope.go:122] "RemoveContainer" containerID="d80ac0e216441dce653df06122088f462ce1b3e86b96173c545c20f98bea455f"
Oct 30 22:04:59.389079 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:04:59.389055    7037 scope.go:122] "RemoveContainer" containerID="942b344308ea9fdf61db11353765aa97e51395f56c421351f41f125c7059c483"
Oct 30 22:05:02.599864 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.599827    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:05:02.603607 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.603590    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.622509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.622459    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d36cd3\"/\"simulator-single-main-d36cd3-kserve-self-signed-certs\""
Oct 30 22:05:02.635164 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.635131    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d36cd3\"/\"default-dockercfg-ff4mn\""
Oct 30 22:05:02.635164 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.635142    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d36cd3\"/\"openshift-service-ca.crt\""
Oct 30 22:05:02.635422 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.635163    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d36cd3\"/\"kube-root-ca.crt\""
Oct 30 22:05:02.637707 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.637680    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:05:02.762779 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.762742    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.762779 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.762780    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.763015 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.762799    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.763015 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.762880    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.763015 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.762918    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.763130 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.763034    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftx4\" (UniqueName: \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864708 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864614    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864708 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864665    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864950 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864709    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864950 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864750    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864950 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864780    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.864950 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.864861    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftx4\" (UniqueName: \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.865208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.865091    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.865208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.865181    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.865292 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.865273    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.866877 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.866860    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.867291 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.867276    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.892597 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.892561    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftx4\" (UniqueName: \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\") pod \"simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") " pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:02.918398 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:02.918359    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:03.245398 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:03.245371    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:05:03.246007 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:05:03.245982    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e9d9bf_3484_4084_8aea_8dc688a43bc9.slice/crio-d91544b20af625a0d7c9292fb48a624341bfe040d03305cebadf19e45a1999f6 WatchSource:0}: Error finding container d91544b20af625a0d7c9292fb48a624341bfe040d03305cebadf19e45a1999f6: Status 404 returned error can't find the container with id d91544b20af625a0d7c9292fb48a624341bfe040d03305cebadf19e45a1999f6
Oct 30 22:05:03.414720 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:03.414675    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" event={"ID":"56e9d9bf-3484-4084-8aea-8dc688a43bc9","Type":"ContainerStarted","Data":"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"}
Oct 30 22:05:03.414920 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:03.414730    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" event={"ID":"56e9d9bf-3484-4084-8aea-8dc688a43bc9","Type":"ContainerStarted","Data":"d91544b20af625a0d7c9292fb48a624341bfe040d03305cebadf19e45a1999f6"}
Oct 30 22:05:03.501985 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:03.501878    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" podStartSLOduration=1.501863319 podStartE2EDuration="1.501863319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:03.50035565 +0000 UTC m=+3846.911895099" watchObservedRunningTime="2026-10-30 22:05:03.501863319 +0000 UTC m=+3846.913402801"
Oct 30 22:05:03.671288 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:05:03.671249    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:05:03.918720 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:03.918686    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:12.918993 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:12.918955    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:12.932198 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:12.932165    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:13.484571 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:13.484544    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:05:26.035593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:26.035552    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"]
Oct 30 22:05:26.047438 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:26.047409    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dwdww"]
Oct 30 22:05:26.706442 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:26.706407    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7635891c-b3c1-4f2e-b830-a57682c03eab" path="/var/lib/kubelet/pods/7635891c-b3c1-4f2e-b830-a57682c03eab/volumes"
Oct 30 22:05:33.677405 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:05:33.677366    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:05:46.027800 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:46.027709    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"]
Oct 30 22:05:46.039845 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:46.039810    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qx5mf"]
Oct 30 22:05:46.707419 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:46.707366    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5bfb5a-590b-49e9-928f-ed79793d5991" path="/var/lib/kubelet/pods/3b5bfb5a-590b-49e9-928f-ed79793d5991/volumes"
Oct 30 22:05:50.111869 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:50.111833    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:05:55.795171 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:55.795133    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:05:58.027909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.027869    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"]
Oct 30 22:05:58.040029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.040001    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qcmsh"]
Oct 30 22:05:58.217952 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.217916    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:05:58.218146 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.218122    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerName="maas-api" containerID="cri-o://563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285" gracePeriod=30
Oct 30 22:05:58.331365 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.331325    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.24:8443/health\": dial tcp 10.134.1.24:8443: connect: connection refused"
Oct 30 22:05:58.673605 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.673571    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:05:58.705433 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.705394    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6652b9b5-4e66-4e9e-915b-6f2da40b98d5" path="/var/lib/kubelet/pods/6652b9b5-4e66-4e9e-915b-6f2da40b98d5/volumes"
Oct 30 22:05:58.728025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.727927    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\") pod \"e3765a30-a8a0-495e-823a-253dcebc53a5\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") "
Oct 30 22:05:58.728208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.728056    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\") pod \"e3765a30-a8a0-495e-823a-253dcebc53a5\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") "
Oct 30 22:05:58.728208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.728117    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\") pod \"e3765a30-a8a0-495e-823a-253dcebc53a5\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") "
Oct 30 22:05:58.728208 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.728206    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\" (UniqueName: \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\") pod \"e3765a30-a8a0-495e-823a-253dcebc53a5\" (UID: \"e3765a30-a8a0-495e-823a-253dcebc53a5\") "
Oct 30 22:05:58.728586 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.728556    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca" pod "e3765a30-a8a0-495e-823a-253dcebc53a5" (UID: "e3765a30-a8a0-495e-823a-253dcebc53a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:58.730122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.730095    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs" pod "e3765a30-a8a0-495e-823a-253dcebc53a5" (UID: "e3765a30-a8a0-495e-823a-253dcebc53a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:58.730262 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.730131    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls" pod "e3765a30-a8a0-495e-823a-253dcebc53a5" (UID: "e3765a30-a8a0-495e-823a-253dcebc53a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:58.730335 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.730264    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz" pod "e3765a30-a8a0-495e-823a-253dcebc53a5" (UID: "e3765a30-a8a0-495e-823a-253dcebc53a5"). InnerVolumeSpecName "kube-api-access-7ptdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:58.741383 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.741353    7037 generic.go:356] "Generic (PLEG): container finished" podID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerID="563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285" exitCode=0
Oct 30 22:05:58.741536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.741404    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"
Oct 30 22:05:58.741536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.741434    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" event={"ID":"e3765a30-a8a0-495e-823a-253dcebc53a5","Type":"ContainerDied","Data":"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"}
Oct 30 22:05:58.741536 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.741532    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8" event={"ID":"e3765a30-a8a0-495e-823a-253dcebc53a5","Type":"ContainerDied","Data":"e3502622a24ce5d561adec1c16047525a08077ce0efeb1da184be123fa90ca5a"}
Oct 30 22:05:58.741739 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.741567    7037 scope.go:122] "RemoveContainer" containerID="563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"
Oct 30 22:05:58.754623 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.754602    7037 scope.go:122] "RemoveContainer" containerID="563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"
Oct 30 22:05:58.754900 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:05:58.754872    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285\": container with ID starting with 563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285 not found: ID does not exist" containerID="563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"
Oct 30 22:05:58.754961 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.754911    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285"} err="failed to get container status \"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285\": rpc error: code = NotFound desc = could not find container \"563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285\": container with ID starting with 563b39375f40ada86efeb1114855c273baabc38fd1740e930b49d25625758285 not found: ID does not exist"
Oct 30 22:05:58.792704 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.792674    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:05:58.798390 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.798363    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d36cd3-7c4754c545-mgpt8"]
Oct 30 22:05:58.830749 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.830526    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:05:58.830749 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.830558    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3765a30-a8a0-495e-823a-253dcebc53a5-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:05:58.830749 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.830573    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3765a30-a8a0-495e-823a-253dcebc53a5-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:05:58.830749 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:58.830588    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptdz\" (UniqueName: \"kubernetes.io/projected/e3765a30-a8a0-495e-823a-253dcebc53a5-kube-api-access-7ptdz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.414529 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:05:59.414472    7037 scope.go:122] "RemoveContainer" containerID="3699a64d3ed2644be43df5c588231c2724af0e7a4332d59c3493c344045fbb03"
Oct 30 22:06:00.714630 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:00.714588    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" path="/var/lib/kubelet/pods/e3765a30-a8a0-495e-823a-253dcebc53a5/volumes"
Oct 30 22:06:00.795590 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:00.795544    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing" containerID="cri-o://7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9" gracePeriod=25
Oct 30 22:06:03.671767 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:06:03.671722    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:06:05.132182 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.132126    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" podUID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" containerName="main" containerID="cri-o://249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc" gracePeriod=45
Oct 30 22:06:05.392113 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.392043    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:06:05.499246 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499214    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499301    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499320    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499370    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499442    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\" (UniqueName: \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499477    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\") pod \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\" (UID: \"56e9d9bf-3484-4084-8aea-8dc688a43bc9\") "
Oct 30 22:06:05.499892 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499554    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:05.499892 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499646    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:05.499892 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499676    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:05.500032 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499893    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.500032 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499919    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.500032 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.499936    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.501709 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.501687    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:05.502120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.502096    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:05.502120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.502106    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4" pod "56e9d9bf-3484-4084-8aea-8dc688a43bc9" (UID: "56e9d9bf-3484-4084-8aea-8dc688a43bc9"). InnerVolumeSpecName "kube-api-access-bftx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:05.600866 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.600834    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftx4\" (UniqueName: \"kubernetes.io/projected/56e9d9bf-3484-4084-8aea-8dc688a43bc9-kube-api-access-bftx4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.600866 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.600865    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e9d9bf-3484-4084-8aea-8dc688a43bc9-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.601077 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.600882    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e9d9bf-3484-4084-8aea-8dc688a43bc9-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:05.782962 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.782870    7037 generic.go:356] "Generic (PLEG): container finished" podID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" containerID="249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc" exitCode=0
Oct 30 22:06:05.782962 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.782929    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"
Oct 30 22:06:05.783180 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.782958    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" event={"ID":"56e9d9bf-3484-4084-8aea-8dc688a43bc9","Type":"ContainerDied","Data":"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"}
Oct 30 22:06:05.783180 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.783000    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd" event={"ID":"56e9d9bf-3484-4084-8aea-8dc688a43bc9","Type":"ContainerDied","Data":"d91544b20af625a0d7c9292fb48a624341bfe040d03305cebadf19e45a1999f6"}
Oct 30 22:06:05.783180 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.783017    7037 scope.go:122] "RemoveContainer" containerID="249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"
Oct 30 22:06:05.792802 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.792780    7037 scope.go:122] "RemoveContainer" containerID="249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"
Oct 30 22:06:05.793073 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:06:05.793050    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc\": container with ID starting with 249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc not found: ID does not exist" containerID="249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"
Oct 30 22:06:05.793155 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.793081    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc"} err="failed to get container status \"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc\": rpc error: code = NotFound desc = could not find container \"249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc\": container with ID starting with 249e3313d50af059bee0d614416101c59ef0df6eb120b486b7cf1699691684cc not found: ID does not exist"
Oct 30 22:06:05.835522 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.835478    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:06:05.837191 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:05.837170    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d36cd3/simulator-single-main-d36cd3-kserve-54c5758b6f-kl4vd"]
Oct 30 22:06:06.705909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.705878    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" path="/var/lib/kubelet/pods/56e9d9bf-3484-4084-8aea-8dc688a43bc9/volumes"
Oct 30 22:06:06.884874 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.884839    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"]
Oct 30 22:06:06.896990 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.896957    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmvn5"]
Oct 30 22:06:06.930349 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930314    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"]
Oct 30 22:06:06.930757 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930745    7037 cpu_manager.go:414] "containerMap: removing container" podUID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" containerName="main"
Oct 30 22:06:06.930814 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930758    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" containerName="main"
Oct 30 22:06:06.930814 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930793    7037 cpu_manager.go:414] "containerMap: removing container" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerName="maas-api"
Oct 30 22:06:06.930814 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930799    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerName="maas-api"
Oct 30 22:06:06.930919 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930894    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3765a30-a8a0-495e-823a-253dcebc53a5" containerName="maas-api"
Oct 30 22:06:06.930919 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.930906    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e9d9bf-3484-4084-8aea-8dc688a43bc9" containerName="main"
Oct 30 22:06:06.935436 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.935413    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:06.950596 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.950563    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:06.950805 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.950637    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:06.950805 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.950759    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:06:06.950956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.950842    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:06.953189 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:06.953155    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"]
Oct 30 22:06:07.015886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.015802    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rwf\" (UniqueName: \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.015886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.015842    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.016074 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.015964    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.117352 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.117317    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.117584 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.117387    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rwf\" (UniqueName: \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.117584 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.117417    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.118025 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.118000    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.119775 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.119754    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.137040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.137015    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rwf\" (UniqueName: \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\") pod \"maas-api-revoke-sub-3d4156f1-h75pg\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.248727 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.248693    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:07.541562 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.541529    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"]
Oct 30 22:06:07.541728 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:06:07.541672    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb13615c_ea38_4dda_b3b4_a07983cfca36.slice/crio-0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878 WatchSource:0}: Error finding container 0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878: Status 404 returned error can't find the container with id 0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878
Oct 30 22:06:07.795530 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:07.795481    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg" event={"ID":"eb13615c-ea38-4dda-b3b4-a07983cfca36","Type":"ContainerStarted","Data":"0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878"}
Oct 30 22:06:08.331324 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:08.331290    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:08.706551 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:08.706466    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca8cf20-c201-4605-9497-efbdf0614a7b" path="/var/lib/kubelet/pods/5ca8cf20-c201-4605-9497-efbdf0614a7b/volumes"
Oct 30 22:06:08.802480 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:08.802448    7037 generic.go:356] "Generic (PLEG): container finished" podID="eb13615c-ea38-4dda-b3b4-a07983cfca36" containerID="c36fcf4092c413fa190f6190501005679f0af33cf21f16d8c880a0fe8251379c" exitCode=0
Oct 30 22:06:08.802901 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:08.802541    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg" event={"ID":"eb13615c-ea38-4dda-b3b4-a07983cfca36","Type":"ContainerDied","Data":"c36fcf4092c413fa190f6190501005679f0af33cf21f16d8c880a0fe8251379c"}
Oct 30 22:06:09.944383 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:09.944358    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:10.047077 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.047043    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\" (UniqueName: \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\") pod \"eb13615c-ea38-4dda-b3b4-a07983cfca36\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") "
Oct 30 22:06:10.047255 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.047083    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\") pod \"eb13615c-ea38-4dda-b3b4-a07983cfca36\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") "
Oct 30 22:06:10.047255 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.047235    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\") pod \"eb13615c-ea38-4dda-b3b4-a07983cfca36\" (UID: \"eb13615c-ea38-4dda-b3b4-a07983cfca36\") "
Oct 30 22:06:10.047640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.047608    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca" pod "eb13615c-ea38-4dda-b3b4-a07983cfca36" (UID: "eb13615c-ea38-4dda-b3b4-a07983cfca36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:10.049203 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.049180    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf" pod "eb13615c-ea38-4dda-b3b4-a07983cfca36" (UID: "eb13615c-ea38-4dda-b3b4-a07983cfca36"). InnerVolumeSpecName "kube-api-access-x5rwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:10.049298 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.049277    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca" pod "eb13615c-ea38-4dda-b3b4-a07983cfca36" (UID: "eb13615c-ea38-4dda-b3b4-a07983cfca36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:10.148833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.148798    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:10.148833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.148827    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5rwf\" (UniqueName: \"kubernetes.io/projected/eb13615c-ea38-4dda-b3b4-a07983cfca36-kube-api-access-x5rwf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:10.148833 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.148841    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb13615c-ea38-4dda-b3b4-a07983cfca36-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:10.817188 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.817151    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"
Oct 30 22:06:10.817366 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.817187    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg" event={"ID":"eb13615c-ea38-4dda-b3b4-a07983cfca36","Type":"ContainerDied","Data":"0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878"}
Oct 30 22:06:10.817366 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:10.817241    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa560d0f8d40f4689a55de60d5679e576152315bb8ee1d9c762f1e575ae4878"
Oct 30 22:06:18.331180 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:18.331136    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:21.049643 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.049613    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:06:21.062709 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.062593    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\") pod \"46720ecb-46bc-4a77-a218-860c7006d18d\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") "
Oct 30 22:06:21.063346 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.062957    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume" pod "46720ecb-46bc-4a77-a218-860c7006d18d" (UID: "46720ecb-46bc-4a77-a218-860c7006d18d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:21.163668 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.163617    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\" (UniqueName: \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\") pod \"46720ecb-46bc-4a77-a218-860c7006d18d\" (UID: \"46720ecb-46bc-4a77-a218-860c7006d18d\") "
Oct 30 22:06:21.163986 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.163966    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46720ecb-46bc-4a77-a218-860c7006d18d-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.165764 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.165734    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r" pod "46720ecb-46bc-4a77-a218-860c7006d18d" (UID: "46720ecb-46bc-4a77-a218-860c7006d18d"). InnerVolumeSpecName "kube-api-access-99v5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:21.264582 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.264545    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99v5r\" (UniqueName: \"kubernetes.io/projected/46720ecb-46bc-4a77-a218-860c7006d18d-kube-api-access-99v5r\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.877776 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.877737    7037 generic.go:356] "Generic (PLEG): container finished" podID="46720ecb-46bc-4a77-a218-860c7006d18d" containerID="7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9" exitCode=0
Oct 30 22:06:21.877974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.877792    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"
Oct 30 22:06:21.877974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.877814    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" event={"ID":"46720ecb-46bc-4a77-a218-860c7006d18d","Type":"ContainerDied","Data":"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"}
Oct 30 22:06:21.877974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.877886    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj" event={"ID":"46720ecb-46bc-4a77-a218-860c7006d18d","Type":"ContainerDied","Data":"cc273912fdcd86212e32b6a0934c31cc3d85a19aebaf552738c7b164c74aec33"}
Oct 30 22:06:21.877974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.877907    7037 scope.go:122] "RemoveContainer" containerID="7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"
Oct 30 22:06:21.890450 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.890430    7037 scope.go:122] "RemoveContainer" containerID="7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"
Oct 30 22:06:21.890708 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:06:21.890690    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9\": container with ID starting with 7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9 not found: ID does not exist" containerID="7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"
Oct 30 22:06:21.890768 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.890714    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9"} err="failed to get container status \"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9\": rpc error: code = NotFound desc = could not find container \"7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9\": container with ID starting with 7c27bf0549964aa6ee8dc10f2ff92a63ca98cadc0ae36e1cddd12122c91b25c9 not found: ID does not exist"
Oct 30 22:06:21.925927 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.925899    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:06:21.936998 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:21.936975    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d36cd3-bb77dv4ljj"]
Oct 30 22:06:22.705997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:22.705965    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" path="/var/lib/kubelet/pods/46720ecb-46bc-4a77-a218-860c7006d18d/volumes"
Oct 30 22:06:27.025751 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:27.025717    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"]
Oct 30 22:06:27.036600 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:27.036570    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-ktgzj"]
Oct 30 22:06:28.705691 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:28.705658    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adb619d-d686-419b-9811-3391e03152c5" path="/var/lib/kubelet/pods/2adb619d-d686-419b-9811-3391e03152c5/volumes"
Oct 30 22:06:33.671370 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:06:33.671328    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:06:37.024554 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:37.024504    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"]
Oct 30 22:06:37.035143 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:37.035108    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jckz7"]
Oct 30 22:06:38.706350 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:38.706309    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00525b43-b221-421c-8027-a1ef120f1c45" path="/var/lib/kubelet/pods/00525b43-b221-421c-8027-a1ef120f1c45/volumes"
Oct 30 22:06:59.025527 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:59.025477    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"]
Oct 30 22:06:59.036207 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:59.036176    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7sb46"]
Oct 30 22:06:59.442232 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:59.442198    7037 scope.go:122] "RemoveContainer" containerID="e2b53d1ecf07ef2782e1c7102313ea0a1fcfee0446143e6e86ce1ff3263907d6"
Oct 30 22:06:59.451108 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:59.451081    7037 scope.go:122] "RemoveContainer" containerID="ca0bc3aa58f63c0235d184f44c6a9cef4ecc93cd1d0a0f6ddcf4ca03ae25baf8"
Oct 30 22:06:59.458944 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:06:59.458922    7037 scope.go:122] "RemoveContainer" containerID="e0b3ed2465e36bda913b26232dc2beb83df3500796032a62f3dbc7febebf0b60"
Oct 30 22:07:00.705372 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:00.705339    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49c3063-65b3-4bfc-b174-846dc3b1532a" path="/var/lib/kubelet/pods/b49c3063-65b3-4bfc-b174-846dc3b1532a/volumes"
Oct 30 22:07:03.671635 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:07:03.671598    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:07:20.023359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:20.023325    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"]
Oct 30 22:07:20.034041 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:20.034011    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-886gw"]
Oct 30 22:07:20.706438 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:20.706404    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc" path="/var/lib/kubelet/pods/28b3ffdf-6d5b-4240-a3cd-faa9c4ca2ddc/volumes"
Oct 30 22:07:33.671652 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:07:33.671612    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:07:41.027865 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:41.027828    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"]
Oct 30 22:07:41.038380 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:41.038345    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lbtr6"]
Oct 30 22:07:42.706170 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:42.706132    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74efb83-7240-4b19-9d41-a5d2138571d7" path="/var/lib/kubelet/pods/e74efb83-7240-4b19-9d41-a5d2138571d7/volumes"
Oct 30 22:07:52.227942 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.227903    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"]
Oct 30 22:07:52.228354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228328    7037 cpu_manager.go:414] "containerMap: removing container" podUID="eb13615c-ea38-4dda-b3b4-a07983cfca36" containerName="revoke-keys"
Oct 30 22:07:52.228354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228339    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb13615c-ea38-4dda-b3b4-a07983cfca36" containerName="revoke-keys"
Oct 30 22:07:52.228436 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228359    7037 cpu_manager.go:414] "containerMap: removing container" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing"
Oct 30 22:07:52.228436 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228365    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing"
Oct 30 22:07:52.228436 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228428    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb13615c-ea38-4dda-b3b4-a07983cfca36" containerName="revoke-keys"
Oct 30 22:07:52.228566 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.228438    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="46720ecb-46bc-4a77-a218-860c7006d18d" containerName="payload-pre-processing"
Oct 30 22:07:52.231922 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.231904    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.258244 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.258214    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:52.258244 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.258210    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:52.258478 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.258279    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:07:52.258478 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.258294    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:52.260150 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.260132    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"]
Oct 30 22:07:52.407132 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.407094    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.407314 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.407215    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.407314 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.407273    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6r4\" (UniqueName: \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.508054 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.507973    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6r4\" (UniqueName: \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.508054 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.508050    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.508397 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.508112    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.508768 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.508743    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.510725 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.510695    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.525659 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.525634    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6r4\" (UniqueName: \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\") pod \"maas-api-revoke-sub-d576e68b-8zmpf\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.545430 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.545403    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:52.835226 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:52.835201    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"]
Oct 30 22:07:52.837197 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:07:52.837160    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e970f63_426f_4540_9f73_1ed3325c99e2.slice/crio-351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9 WatchSource:0}: Error finding container 351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9: Status 404 returned error can't find the container with id 351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9
Oct 30 22:07:53.379392 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:53.379356    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf" event={"ID":"2e970f63-426f-4540-9f73-1ed3325c99e2","Type":"ContainerStarted","Data":"a9bf2e57e0310f9df0a8b55cd1150c61b7633f43b1b6de7f1d7dc7d105a8567b"}
Oct 30 22:07:53.379806 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:53.379418    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf" event={"ID":"2e970f63-426f-4540-9f73-1ed3325c99e2","Type":"ContainerStarted","Data":"351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9"}
Oct 30 22:07:53.416627 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:53.416536    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf" podStartSLOduration=1.416522738 podStartE2EDuration="1.416522738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:53.4146436 +0000 UTC m=+4016.826183032" watchObservedRunningTime="2026-10-30 22:07:53.416522738 +0000 UTC m=+4016.828062175"
Oct 30 22:07:54.386345 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:54.386304    7037 generic.go:356] "Generic (PLEG): container finished" podID="2e970f63-426f-4540-9f73-1ed3325c99e2" containerID="a9bf2e57e0310f9df0a8b55cd1150c61b7633f43b1b6de7f1d7dc7d105a8567b" exitCode=0
Oct 30 22:07:54.386765 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:54.386384    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf" event={"ID":"2e970f63-426f-4540-9f73-1ed3325c99e2","Type":"ContainerDied","Data":"a9bf2e57e0310f9df0a8b55cd1150c61b7633f43b1b6de7f1d7dc7d105a8567b"}
Oct 30 22:07:55.526090 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.526057    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:55.640722 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.640685    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\") pod \"2e970f63-426f-4540-9f73-1ed3325c99e2\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") "
Oct 30 22:07:55.640911 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.640730    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\") pod \"2e970f63-426f-4540-9f73-1ed3325c99e2\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") "
Oct 30 22:07:55.640911 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.640780    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\" (UniqueName: \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\") pod \"2e970f63-426f-4540-9f73-1ed3325c99e2\" (UID: \"2e970f63-426f-4540-9f73-1ed3325c99e2\") "
Oct 30 22:07:55.641108 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.641084    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca" pod "2e970f63-426f-4540-9f73-1ed3325c99e2" (UID: "2e970f63-426f-4540-9f73-1ed3325c99e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:55.642899 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.642836    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca" pod "2e970f63-426f-4540-9f73-1ed3325c99e2" (UID: "2e970f63-426f-4540-9f73-1ed3325c99e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:55.642994 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.642904    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4" pod "2e970f63-426f-4540-9f73-1ed3325c99e2" (UID: "2e970f63-426f-4540-9f73-1ed3325c99e2"). InnerVolumeSpecName "kube-api-access-xs6r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:55.742407 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.742366    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:07:55.742407 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.742404    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e970f63-426f-4540-9f73-1ed3325c99e2-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:07:55.742626 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:55.742417    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6r4\" (UniqueName: \"kubernetes.io/projected/2e970f63-426f-4540-9f73-1ed3325c99e2-kube-api-access-xs6r4\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:07:56.398407 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:56.398363    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf" event={"ID":"2e970f63-426f-4540-9f73-1ed3325c99e2","Type":"ContainerDied","Data":"351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9"}
Oct 30 22:07:56.398617 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:56.398422    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351f9bedddcfefb3fc57b3bab4ea7bd4abecdc95c1ec9ed980449c9679d26fa9"
Oct 30 22:07:56.398617 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:56.398385    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"
Oct 30 22:07:58.026695 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:58.026663    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"]
Oct 30 22:07:58.037577 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:58.037548    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-59svf"]
Oct 30 22:07:58.705374 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:58.705341    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e80508b-898f-434d-808d-be1fe3ed5986" path="/var/lib/kubelet/pods/0e80508b-898f-434d-808d-be1fe3ed5986/volumes"
Oct 30 22:07:59.497192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:59.497144    7037 scope.go:122] "RemoveContainer" containerID="48b710236c04868cc714297f1ccec8419a0364e4b70c2d9ec57c6fbfc318a040"
Oct 30 22:07:59.506037 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:59.506013    7037 scope.go:122] "RemoveContainer" containerID="81a989b3637ec26cc115ee0f1ca510a065092802ee4dac458b6d8b1e66d89219"
Oct 30 22:07:59.513937 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:07:59.513906    7037 scope.go:122] "RemoveContainer" containerID="8dd7c8d22dd51a8a28c11f91810a32ccc933e8d010a0f1556b29cc8e45e9d7bd"
Oct 30 22:08:03.671671 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:08:03.671633    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:08:13.427483 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.427442    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"]
Oct 30 22:08:13.428004 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.427908    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2e970f63-426f-4540-9f73-1ed3325c99e2" containerName="revoke-keys"
Oct 30 22:08:13.428004 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.427921    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e970f63-426f-4540-9f73-1ed3325c99e2" containerName="revoke-keys"
Oct 30 22:08:13.428004 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.427990    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e970f63-426f-4540-9f73-1ed3325c99e2" containerName="revoke-keys"
Oct 30 22:08:13.430932 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.430910    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.446904 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.446878    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:13.457678 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.457651    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:13.457881 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.457682    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:08:13.457881 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.457743    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:13.459628 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.459606    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"]
Oct 30 22:08:13.553817 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.553785    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.553998 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.553859    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.553998 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.553931    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhqp\" (UniqueName: \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.655000 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.654963    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.655197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.655048    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.655197 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.655078    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhqp\" (UniqueName: \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.655659 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.655637    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.657414 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.657396    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.673878 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.673852    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhqp\" (UniqueName: \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\") pod \"maas-api-revoke-sub-d2bc93bc-zm6c4\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:13.744020 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:13.743926    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:14.043811 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:14.043726    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"]
Oct 30 22:08:14.047594 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:08:14.047561    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2328a09c_e579_41f5_aa43_a636a0ec0cb7.slice/crio-6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9 WatchSource:0}: Error finding container 6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9: Status 404 returned error can't find the container with id 6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9
Oct 30 22:08:14.527159 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:14.527125    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4" event={"ID":"2328a09c-e579-41f5-aa43-a636a0ec0cb7","Type":"ContainerStarted","Data":"fb30f81746c179a8c0476a0944bef4c7a1dff6bdfce778133cd9fa70ff23ec3e"}
Oct 30 22:08:14.527587 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:14.527184    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4" event={"ID":"2328a09c-e579-41f5-aa43-a636a0ec0cb7","Type":"ContainerStarted","Data":"6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9"}
Oct 30 22:08:14.564849 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:14.564804    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4" podStartSLOduration=1.5647912609999999 podStartE2EDuration="1.564791261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:14.56305813 +0000 UTC m=+4037.974597570" watchObservedRunningTime="2026-10-30 22:08:14.564791261 +0000 UTC m=+4037.976330695"
Oct 30 22:08:15.534160 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:15.534125    7037 generic.go:356] "Generic (PLEG): container finished" podID="2328a09c-e579-41f5-aa43-a636a0ec0cb7" containerID="fb30f81746c179a8c0476a0944bef4c7a1dff6bdfce778133cd9fa70ff23ec3e" exitCode=0
Oct 30 22:08:15.534582 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:15.534208    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4" event={"ID":"2328a09c-e579-41f5-aa43-a636a0ec0cb7","Type":"ContainerDied","Data":"fb30f81746c179a8c0476a0944bef4c7a1dff6bdfce778133cd9fa70ff23ec3e"}
Oct 30 22:08:16.666697 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.666673    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:16.784893 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.784863    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\") pod \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") "
Oct 30 22:08:16.785093 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.784916    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\") pod \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") "
Oct 30 22:08:16.785093 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.784943    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\" (UniqueName: \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\") pod \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\" (UID: \"2328a09c-e579-41f5-aa43-a636a0ec0cb7\") "
Oct 30 22:08:16.785368 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.785343    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca" pod "2328a09c-e579-41f5-aa43-a636a0ec0cb7" (UID: "2328a09c-e579-41f5-aa43-a636a0ec0cb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:16.787044 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.787023    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp" pod "2328a09c-e579-41f5-aa43-a636a0ec0cb7" (UID: "2328a09c-e579-41f5-aa43-a636a0ec0cb7"). InnerVolumeSpecName "kube-api-access-fbhqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:16.787130 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.787085    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca" pod "2328a09c-e579-41f5-aa43-a636a0ec0cb7" (UID: "2328a09c-e579-41f5-aa43-a636a0ec0cb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:16.886564 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.886526    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:08:16.886564 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.886558    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2328a09c-e579-41f5-aa43-a636a0ec0cb7-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:08:16.886564 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:16.886568    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhqp\" (UniqueName: \"kubernetes.io/projected/2328a09c-e579-41f5-aa43-a636a0ec0cb7-kube-api-access-fbhqp\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:08:17.547632 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:17.547598    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"
Oct 30 22:08:17.547632 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:17.547621    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4" event={"ID":"2328a09c-e579-41f5-aa43-a636a0ec0cb7","Type":"ContainerDied","Data":"6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9"}
Oct 30 22:08:17.547839 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:17.547672    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a9c5c33352d8e2014ac66e98701198a292ab85d2f6da0363f0d5b11023979a9"
Oct 30 22:08:31.078773 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.078732    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:08:31.081364 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.079171    7037 cpu_manager.go:414] "containerMap: removing container" podUID="2328a09c-e579-41f5-aa43-a636a0ec0cb7" containerName="revoke-keys"
Oct 30 22:08:31.081364 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.079187    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2328a09c-e579-41f5-aa43-a636a0ec0cb7" containerName="revoke-keys"
Oct 30 22:08:31.081364 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.079290    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="2328a09c-e579-41f5-aa43-a636a0ec0cb7" containerName="revoke-keys"
Oct 30 22:08:31.082464 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.082444    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.108791 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.108757    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:31.108791 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.108786    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:31.109009 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.108776    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9b97ce-serving-cert\""
Oct 30 22:08:31.109009 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.108776    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pdl4\""
Oct 30 22:08:31.109009 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.108936    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:08:31.111257 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.111233    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:08:31.134687 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.134653    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.134859 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.134709    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.134859 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.134814    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m4h\" (UniqueName: \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.134957 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.134859    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.235848 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.235805    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.236057 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.235885    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.236057 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.235929    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m4h\" (UniqueName: \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.236057 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.235981    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.236600 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.236575    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.238259 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.238231    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.238481 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.238459    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.253967 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.253936    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m4h\" (UniqueName: \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\") pod \"maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.395626 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.395538    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:31.507403 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.507364    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:08:31.512110 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.512089    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.530862 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.530838    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:08:31.531800 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.531774    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-093fe422-dockercfg-5j9rr\""
Oct 30 22:08:31.531920 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.531838    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9b97ce\""
Oct 30 22:08:31.538573 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.538549    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkw4g\" (UniqueName: \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.538673 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.538590    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.614029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.613997    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:08:31.619140 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.619074    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.636085 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.636052    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:08:31.639684 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.639656    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.639855 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.639719    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxn4k\" (UniqueName: \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.639944 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.639866    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkw4g\" (UniqueName: \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.639944 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.639910    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.640792 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.640768    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.660788 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.660733    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkw4g\" (UniqueName: \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\") pod \"payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.709174 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.709139    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:08:31.713650 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:08:31.713624    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125a17e1_0b67_42e4_aec5_6ccbf11b5e96.slice/crio-e8991b05b066815cb7a7a38a2d06ceeb9ec2b7fc6570228a0416b53c404c270f WatchSource:0}: Error finding container e8991b05b066815cb7a7a38a2d06ceeb9ec2b7fc6570228a0416b53c404c270f: Status 404 returned error can't find the container with id e8991b05b066815cb7a7a38a2d06ceeb9ec2b7fc6570228a0416b53c404c270f
Oct 30 22:08:31.741090 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.741053    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.741205 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.741125    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxn4k\" (UniqueName: \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.741784 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.741763    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.758804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.758783    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxn4k\" (UniqueName: \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\") pod \"payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:31.825664 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.825627    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:31.938168 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:31.938095    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:32.129281 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.129246    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:08:32.136958 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:08:32.136926    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f71a49_415e_45ad_b02a_93873e9b6202.slice/crio-b4c09f5281b8e09a5f84c6610af7cd726f7f1bfb8f5004f26b74e4c3bcf5f61f WatchSource:0}: Error finding container b4c09f5281b8e09a5f84c6610af7cd726f7f1bfb8f5004f26b74e4c3bcf5f61f: Status 404 returned error can't find the container with id b4c09f5281b8e09a5f84c6610af7cd726f7f1bfb8f5004f26b74e4c3bcf5f61f
Oct 30 22:08:32.233288 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.233200    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:08:32.237033 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:08:32.237001    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782faf31_28f2_4da6_ad1e_521e67b0be5b.slice/crio-322c593e846d0ab383a3b6836bd96d86685254906d952967b6aba852d195dbe0 WatchSource:0}: Error finding container 322c593e846d0ab383a3b6836bd96d86685254906d952967b6aba852d195dbe0: Status 404 returned error can't find the container with id 322c593e846d0ab383a3b6836bd96d86685254906d952967b6aba852d195dbe0
Oct 30 22:08:32.629678 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.629633    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" event={"ID":"782faf31-28f2-4da6-ad1e-521e67b0be5b","Type":"ContainerStarted","Data":"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"}
Oct 30 22:08:32.629888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.629702    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" event={"ID":"782faf31-28f2-4da6-ad1e-521e67b0be5b","Type":"ContainerStarted","Data":"322c593e846d0ab383a3b6836bd96d86685254906d952967b6aba852d195dbe0"}
Oct 30 22:08:32.629888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.629822    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:32.631252 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.631224    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" event={"ID":"125a17e1-0b67-42e4-aec5-6ccbf11b5e96","Type":"ContainerStarted","Data":"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"}
Oct 30 22:08:32.631381 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.631269    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" event={"ID":"125a17e1-0b67-42e4-aec5-6ccbf11b5e96","Type":"ContainerStarted","Data":"e8991b05b066815cb7a7a38a2d06ceeb9ec2b7fc6570228a0416b53c404c270f"}
Oct 30 22:08:32.631381 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.631307    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:32.632816 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.632788    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" event={"ID":"74f71a49-415e-45ad-b02a-93873e9b6202","Type":"ContainerStarted","Data":"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"}
Oct 30 22:08:32.633143 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.632836    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" event={"ID":"74f71a49-415e-45ad-b02a-93873e9b6202","Type":"ContainerStarted","Data":"b4c09f5281b8e09a5f84c6610af7cd726f7f1bfb8f5004f26b74e4c3bcf5f61f"}
Oct 30 22:08:32.633143 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.632911    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:32.670385 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.670334    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" podStartSLOduration=1.67032 podStartE2EDuration="1.67032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.668926725 +0000 UTC m=+4056.080466165" watchObservedRunningTime="2026-10-30 22:08:32.67032 +0000 UTC m=+4056.081859428"
Oct 30 22:08:32.711371 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.711314    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" podStartSLOduration=1.319224143 podStartE2EDuration="1.711299772s" totalImagesPullingTime="392.075629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.710369482 +0000 UTC m=+4056.121908945" watchObservedRunningTime="2026-10-30 22:08:32.711299772 +0000 UTC m=+4056.122839209"
Oct 30 22:08:32.746617 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:32.746558    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" podStartSLOduration=1.7465434119999999 podStartE2EDuration="1.746543412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.745114758 +0000 UTC m=+4056.156654199" watchObservedRunningTime="2026-10-30 22:08:32.746543412 +0000 UTC m=+4056.158082851"
Oct 30 22:08:33.672614 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:08:33.672579    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:08:38.642364 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:38.642282    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:08:43.640818 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:43.640783    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:08:43.641318 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:43.640872    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:08:59.031622 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:59.031588    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"]
Oct 30 22:08:59.043578 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:59.043036    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-75gn2"]
Oct 30 22:08:59.542040 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:59.542000    7037 scope.go:122] "RemoveContainer" containerID="e9dd84ccd679038e9230a45a1a866ba805d673d6d02b7eefae83561ba454b186"
Oct 30 22:08:59.551045 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:59.551025    7037 scope.go:122] "RemoveContainer" containerID="e58235692b6d64986ed2663aa50af95180d74330f05f9f3c15bab4638ea72353"
Oct 30 22:08:59.559096 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:08:59.559071    7037 scope.go:122] "RemoveContainer" containerID="1a5a69de59352be0c3349cff6e932259cb97e07443b5aacb96e3b4b39a6c0559"
Oct 30 22:09:00.705908 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:00.705874    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19cd566-3169-4547-b70e-7306a9eb1aa5" path="/var/lib/kubelet/pods/f19cd566-3169-4547-b70e-7306a9eb1aa5/volumes"
Oct 30 22:09:03.670588 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:09:03.670552    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:09:09.025014 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:09.024978    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"]
Oct 30 22:09:09.035675 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:09.035652    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2g7lg"]
Oct 30 22:09:10.705796 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:10.705760    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16723ad6-5174-4368-8230-9bc885f227b3" path="/var/lib/kubelet/pods/16723ad6-5174-4368-8230-9bc885f227b3/volumes"
Oct 30 22:09:33.671680 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:09:33.671644    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:09:38.515132 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:38.515092    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:09:38.546328 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:38.546292    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:09:40.023555 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.023521    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:09:40.024076 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.023740    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" podUID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" containerName="maas-api" containerID="cri-o://24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b" gracePeriod=30
Oct 30 22:09:40.484047 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.484020    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:09:40.634812 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.634777    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\") pod \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") "
Oct 30 22:09:40.635035 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.634921    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\") pod \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") "
Oct 30 22:09:40.635111 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.635036    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\" (UniqueName: \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\") pod \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") "
Oct 30 22:09:40.635111 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.635068    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\") pod \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\" (UID: \"125a17e1-0b67-42e4-aec5-6ccbf11b5e96\") "
Oct 30 22:09:40.635293 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.635263    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca" pod "125a17e1-0b67-42e4-aec5-6ccbf11b5e96" (UID: "125a17e1-0b67-42e4-aec5-6ccbf11b5e96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:40.635449 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.635432    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:09:40.637125 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.637100    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs" pod "125a17e1-0b67-42e4-aec5-6ccbf11b5e96" (UID: "125a17e1-0b67-42e4-aec5-6ccbf11b5e96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:40.637251 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.637201    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h" pod "125a17e1-0b67-42e4-aec5-6ccbf11b5e96" (UID: "125a17e1-0b67-42e4-aec5-6ccbf11b5e96"). InnerVolumeSpecName "kube-api-access-t6m4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:40.637348 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.637322    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls" pod "125a17e1-0b67-42e4-aec5-6ccbf11b5e96" (UID: "125a17e1-0b67-42e4-aec5-6ccbf11b5e96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:40.736888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.736857    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6m4h\" (UniqueName: \"kubernetes.io/projected/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-kube-api-access-t6m4h\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:09:40.736888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.736883    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:09:40.736888 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:40.736892    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125a17e1-0b67-42e4-aec5-6ccbf11b5e96-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:09:41.020991 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.020904    7037 generic.go:356] "Generic (PLEG): container finished" podID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" containerID="24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b" exitCode=0
Oct 30 22:09:41.020991 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.020963    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"
Oct 30 22:09:41.021192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.020983    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" event={"ID":"125a17e1-0b67-42e4-aec5-6ccbf11b5e96","Type":"ContainerDied","Data":"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"}
Oct 30 22:09:41.021192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.021060    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb" event={"ID":"125a17e1-0b67-42e4-aec5-6ccbf11b5e96","Type":"ContainerDied","Data":"e8991b05b066815cb7a7a38a2d06ceeb9ec2b7fc6570228a0416b53c404c270f"}
Oct 30 22:09:41.021192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.021083    7037 scope.go:122] "RemoveContainer" containerID="24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"
Oct 30 22:09:41.030540 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.030274    7037 scope.go:122] "RemoveContainer" containerID="24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"
Oct 30 22:09:41.030821 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:09:41.030648    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b\": container with ID starting with 24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b not found: ID does not exist" containerID="24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"
Oct 30 22:09:41.030821 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.030688    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b"} err="failed to get container status \"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b\": rpc error: code = NotFound desc = could not find container \"24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b\": container with ID starting with 24219d59d7e167a9eecd401b4c1ae6d1482b5613b49e483cf0297893fe1eb43b not found: ID does not exist"
Oct 30 22:09:41.067428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.067399    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:09:41.078020 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:41.077995    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9b97ce-858f66fb57-nprmb"]
Oct 30 22:09:42.706746 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:42.706713    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" path="/var/lib/kubelet/pods/125a17e1-0b67-42e4-aec5-6ccbf11b5e96/volumes"
Oct 30 22:09:43.516042 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:43.516003    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing" containerID="cri-o://0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08" gracePeriod=25
Oct 30 22:09:43.547579 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:43.547536    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing" containerID="cri-o://94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3" gracePeriod=25
Oct 30 22:09:43.638858 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:43.638824    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:43.638858 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:43.638847    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:53.638789 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.638745    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:53.639210 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.638778    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:53.765907 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.765868    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:09:53.766509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.766467    7037 cpu_manager.go:414] "containerMap: removing container" podUID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" containerName="maas-api"
Oct 30 22:09:53.766509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.766505    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" containerName="maas-api"
Oct 30 22:09:53.767089 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.767061    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="125a17e1-0b67-42e4-aec5-6ccbf11b5e96" containerName="maas-api"
Oct 30 22:09:53.781311 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.781280    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.788704 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.788676    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:09:53.796654 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.796628    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-893c35ed-kserve-self-signed-certs\""
Oct 30 22:09:53.867543 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867506    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.867726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867633    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.867726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867677    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.867726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867704    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfwh\" (UniqueName: \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.867851 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867761    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.867851 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.867831    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.968713 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968605    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfwh\" (UniqueName: \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.968713 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968695    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968733    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968795    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968905    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.968959    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.969393    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.969421    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.970781 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.969478    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.971826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.971801    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.971826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.971813    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:53.986181 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:53.986156    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfwh\" (UniqueName: \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\") pod \"e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") " pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:54.095633 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:54.095603    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:54.391052 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:54.391021    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:09:54.395088 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:09:54.395052    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4edddff_2d2b_4918_a86c_ceb91188699c.slice/crio-f629d71f6fd3960f829b2df23ed41dd35738a1fb3f4fe91e6f703b97390adaea WatchSource:0}: Error finding container f629d71f6fd3960f829b2df23ed41dd35738a1fb3f4fe91e6f703b97390adaea: Status 404 returned error can't find the container with id f629d71f6fd3960f829b2df23ed41dd35738a1fb3f4fe91e6f703b97390adaea
Oct 30 22:09:55.098472 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:55.098435    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" event={"ID":"a4edddff-2d2b-4918-a86c-ceb91188699c","Type":"ContainerStarted","Data":"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"}
Oct 30 22:09:55.098472 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:55.098475    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" event={"ID":"a4edddff-2d2b-4918-a86c-ceb91188699c","Type":"ContainerStarted","Data":"f629d71f6fd3960f829b2df23ed41dd35738a1fb3f4fe91e6f703b97390adaea"}
Oct 30 22:09:55.136569 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:55.136513    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" podStartSLOduration=2.136479662 podStartE2EDuration="2.136479662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:55.135739383 +0000 UTC m=+4138.547278822" watchObservedRunningTime="2026-10-30 22:09:55.136479662 +0000 UTC m=+4138.548019100"
Oct 30 22:09:56.096409 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:56.096372    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:09:59.586344 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:59.586312    7037 scope.go:122] "RemoveContainer" containerID="fef83670fe2bae8cb9f81281d20712778b27e7fdaabc47298ac5ee9df9c2a079"
Oct 30 22:09:59.595431 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:59.595403    7037 scope.go:122] "RemoveContainer" containerID="061cba0c72da798c635001a06a66d7b8c2aa69effc6f897cd1ca559280526f01"
Oct 30 22:09:59.605792 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:09:59.605768    7037 scope.go:122] "RemoveContainer" containerID="a4a4191ebc41400a102f44b9dcc66c81d63bd151d9f67763f5479f4966dbd299"
Oct 30 22:10:03.640754 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:10:03.640729    7037 logging.go:55] [core] [Channel #17093 SubChannel #17095]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.31:50052", ServerName: "10.134.1.31:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.31:50052: connect: connection refused"
Oct 30 22:10:03.641095 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:10:03.640730    7037 logging.go:55] [core] [Channel #17094 SubChannel #17096]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.32:50052", ServerName: "10.134.1.32:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.32:50052: connect: connection refused"
Oct 30 22:10:03.679035 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:03.678997    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:10:03.823662 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.823637    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:10:03.826850 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.826827    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:10:03.873871 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.873845    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\" (UniqueName: \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\") pod \"74f71a49-415e-45ad-b02a-93873e9b6202\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") "
Oct 30 22:10:03.874010 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.873889    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\" (UniqueName: \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\") pod \"782faf31-28f2-4da6-ad1e-521e67b0be5b\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") "
Oct 30 22:10:03.874010 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.873956    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\") pod \"782faf31-28f2-4da6-ad1e-521e67b0be5b\" (UID: \"782faf31-28f2-4da6-ad1e-521e67b0be5b\") "
Oct 30 22:10:03.874010 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.873983    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\") pod \"74f71a49-415e-45ad-b02a-93873e9b6202\" (UID: \"74f71a49-415e-45ad-b02a-93873e9b6202\") "
Oct 30 22:10:03.874368 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.874332    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume" pod "782faf31-28f2-4da6-ad1e-521e67b0be5b" (UID: "782faf31-28f2-4da6-ad1e-521e67b0be5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:03.874537 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.874382    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume" pod "74f71a49-415e-45ad-b02a-93873e9b6202" (UID: "74f71a49-415e-45ad-b02a-93873e9b6202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:03.876103 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.876075    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k" pod "782faf31-28f2-4da6-ad1e-521e67b0be5b" (UID: "782faf31-28f2-4da6-ad1e-521e67b0be5b"). InnerVolumeSpecName "kube-api-access-vxn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:03.876195 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.876085    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g" pod "74f71a49-415e-45ad-b02a-93873e9b6202" (UID: "74f71a49-415e-45ad-b02a-93873e9b6202"). InnerVolumeSpecName "kube-api-access-vkw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:03.974968 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.974932    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkw4g\" (UniqueName: \"kubernetes.io/projected/74f71a49-415e-45ad-b02a-93873e9b6202-kube-api-access-vkw4g\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:03.974968 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.974965    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxn4k\" (UniqueName: \"kubernetes.io/projected/782faf31-28f2-4da6-ad1e-521e67b0be5b-kube-api-access-vxn4k\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:03.975192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.974977    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782faf31-28f2-4da6-ad1e-521e67b0be5b-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:03.975192 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:03.974990    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f71a49-415e-45ad-b02a-93873e9b6202-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:04.096659 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.096623    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:10:04.109241 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.109217    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:10:04.151387 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.151355    7037 generic.go:356] "Generic (PLEG): container finished" podID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerID="0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08" exitCode=0
Oct 30 22:10:04.151568 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.151406    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"
Oct 30 22:10:04.151568 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.151442    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" event={"ID":"782faf31-28f2-4da6-ad1e-521e67b0be5b","Type":"ContainerDied","Data":"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"}
Oct 30 22:10:04.151568 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.151523    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" event={"ID":"782faf31-28f2-4da6-ad1e-521e67b0be5b","Type":"ContainerDied","Data":"322c593e846d0ab383a3b6836bd96d86685254906d952967b6aba852d195dbe0"}
Oct 30 22:10:04.151568 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.151550    7037 scope.go:122] "RemoveContainer" containerID="0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"
Oct 30 22:10:04.152882 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.152858    7037 generic.go:356] "Generic (PLEG): container finished" podID="74f71a49-415e-45ad-b02a-93873e9b6202" containerID="94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3" exitCode=0
Oct 30 22:10:04.152994 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.152897    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"
Oct 30 22:10:04.152994 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.152953    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" event={"ID":"74f71a49-415e-45ad-b02a-93873e9b6202","Type":"ContainerDied","Data":"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"}
Oct 30 22:10:04.153136 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.153004    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" event={"ID":"74f71a49-415e-45ad-b02a-93873e9b6202","Type":"ContainerDied","Data":"b4c09f5281b8e09a5f84c6610af7cd726f7f1bfb8f5004f26b74e4c3bcf5f61f"}
Oct 30 22:10:04.161810 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.161788    7037 scope.go:122] "RemoveContainer" containerID="0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"
Oct 30 22:10:04.162061 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:04.162040    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08\": container with ID starting with 0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08 not found: ID does not exist" containerID="0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"
Oct 30 22:10:04.162109 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.162068    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08"} err="failed to get container status \"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08\": rpc error: code = NotFound desc = could not find container \"0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08\": container with ID starting with 0a5880e1471405ec3d37802303b9d641e7e5cd3d7846df586a51a2101d267b08 not found: ID does not exist"
Oct 30 22:10:04.162109 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.162084    7037 scope.go:122] "RemoveContainer" containerID="94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"
Oct 30 22:10:04.170991 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.170962    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:10:04.173293 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.173275    7037 scope.go:122] "RemoveContainer" containerID="94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"
Oct 30 22:10:04.173549 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:04.173527    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3\": container with ID starting with 94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3 not found: ID does not exist" containerID="94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"
Oct 30 22:10:04.173625 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.173560    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3"} err="failed to get container status \"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3\": rpc error: code = NotFound desc = could not find container \"94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3\": container with ID starting with 94f59794acb2b4b68ba2ed25c7504b92910936ef39b5fb00b1b88a7572ed49f3 not found: ID does not exist"
Oct 30 22:10:04.197526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.197483    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:10:04.201480 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.201456    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh"]
Oct 30 22:10:04.233383 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.233354    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:10:04.237934 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.237906    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw"]
Oct 30 22:10:04.639343 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.639300    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9b97ce-6f9f9lg6fh" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.31:50052\" within 1s: context deadline exceeded"
Oct 30 22:10:04.639633 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.639348    7037 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b97ce-56bdc8f9c8jnrw" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.32:50052\" within 1s: context deadline exceeded"
Oct 30 22:10:04.709829 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.709786    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" path="/var/lib/kubelet/pods/74f71a49-415e-45ad-b02a-93873e9b6202/volumes"
Oct 30 22:10:04.710404 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:04.710383    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" path="/var/lib/kubelet/pods/782faf31-28f2-4da6-ad1e-521e67b0be5b/volumes"
Oct 30 22:10:08.397287 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.397199    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:08.398476 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398443    7037 cpu_manager.go:414] "containerMap: removing container" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing"
Oct 30 22:10:08.398476 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398476    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing"
Oct 30 22:10:08.398683 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398550    7037 cpu_manager.go:414] "containerMap: removing container" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing"
Oct 30 22:10:08.398683 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398560    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing"
Oct 30 22:10:08.398804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398699    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="782faf31-28f2-4da6-ad1e-521e67b0be5b" containerName="payload-processing"
Oct 30 22:10:08.398804 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.398720    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f71a49-415e-45ad-b02a-93873e9b6202" containerName="payload-pre-processing"
Oct 30 22:10:08.403401 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.403378    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.433000 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.432957    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-893c35ed-kserve-self-signed-certs\""
Oct 30 22:10:08.434600 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.434577    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:08.517666 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517639    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.517837 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517671    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2knf\" (UniqueName: \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.517837 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517715    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.517932 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517845    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.517932 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517881    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.518016 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.517970    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619547 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619511    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619547 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619552    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2knf\" (UniqueName: \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619602    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619650    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619672    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.619826 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.619729    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.620060 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.620006    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.620122 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.620063    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.620186 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.620146    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.621897 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.621880    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.622365 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.622342    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.638039 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.638003    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2knf\" (UniqueName: \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\") pod \"e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") " pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:08.716057 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:08.715993    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:09.009842 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:09.009766    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:09.013560 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:10:09.013531    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeca6b3ad_29d0_4c55_b808_c6f216a10f0b.slice/crio-e19d31953001527e3a27d6f8e32a38105f38f4f0486206e8ca1e6607586aa5a5 WatchSource:0}: Error finding container e19d31953001527e3a27d6f8e32a38105f38f4f0486206e8ca1e6607586aa5a5: Status 404 returned error can't find the container with id e19d31953001527e3a27d6f8e32a38105f38f4f0486206e8ca1e6607586aa5a5
Oct 30 22:10:09.182922 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:09.182876    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" event={"ID":"eca6b3ad-29d0-4c55-b808-c6f216a10f0b","Type":"ContainerStarted","Data":"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"}
Oct 30 22:10:09.182922 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:09.182925    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" event={"ID":"eca6b3ad-29d0-4c55-b808-c6f216a10f0b","Type":"ContainerStarted","Data":"e19d31953001527e3a27d6f8e32a38105f38f4f0486206e8ca1e6607586aa5a5"}
Oct 30 22:10:09.221398 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:09.221340    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" podStartSLOduration=1.22132199 podStartE2EDuration="1.22132199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:09.220082347 +0000 UTC m=+4152.631621784" watchObservedRunningTime="2026-10-30 22:10:09.22132199 +0000 UTC m=+4152.632861428"
Oct 30 22:10:09.716912 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:09.716877    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:18.716752 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:18.716716    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:18.729505 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:18.729460    7037 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:19.248333 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:19.248299    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:25.106256 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.106215    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"]
Oct 30 22:10:25.110265 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.110244    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.125934 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.125904    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:25.136811 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.136775    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:25.136952 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.136835    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:10:25.136952 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.136847    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:25.138719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.138699    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"]
Oct 30 22:10:25.218838 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.218804    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.219016 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.218852    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd6s\" (UniqueName: \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.219058 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.219022    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.319868 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.319834    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.320112 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.319901    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.320112 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.319935    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd6s\" (UniqueName: \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.320595 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.320567    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.322324 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.322301    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.337950 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.337923    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd6s\" (UniqueName: \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\") pod \"maas-api-revoke-sub-452b6d7b-shgc9\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.424107 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.424039    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:25.719989 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:25.719914    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"]
Oct 30 22:10:25.723905 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:10:25.723876    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401d6f8c_5480_410d_99a4_f81f4ffb2d8b.slice/crio-dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65 WatchSource:0}: Error finding container dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65: Status 404 returned error can't find the container with id dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65
Oct 30 22:10:26.276719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:26.276682    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9" event={"ID":"401d6f8c-5480-410d-99a4-f81f4ffb2d8b","Type":"ContainerStarted","Data":"587c7c66144fe2708ff5900e5e42707e1a975fdef18812f8a7d4199ad510e935"}
Oct 30 22:10:26.277133 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:26.276739    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9" event={"ID":"401d6f8c-5480-410d-99a4-f81f4ffb2d8b","Type":"ContainerStarted","Data":"dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65"}
Oct 30 22:10:26.312736 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:26.312632    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9" podStartSLOduration=1.3126171580000001 podStartE2EDuration="1.312617158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:26.311833971 +0000 UTC m=+4169.723373412" watchObservedRunningTime="2026-10-30 22:10:26.312617158 +0000 UTC m=+4169.724156592"
Oct 30 22:10:27.286997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:27.286947    7037 generic.go:356] "Generic (PLEG): container finished" podID="401d6f8c-5480-410d-99a4-f81f4ffb2d8b" containerID="587c7c66144fe2708ff5900e5e42707e1a975fdef18812f8a7d4199ad510e935" exitCode=0
Oct 30 22:10:27.287664 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:27.287018    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9" event={"ID":"401d6f8c-5480-410d-99a4-f81f4ffb2d8b","Type":"ContainerDied","Data":"587c7c66144fe2708ff5900e5e42707e1a975fdef18812f8a7d4199ad510e935"}
Oct 30 22:10:28.723327 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.723302    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:28.810521 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.810468    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\" (UniqueName: \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\") pod \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") "
Oct 30 22:10:28.811073 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.810993    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\") pod \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") "
Oct 30 22:10:28.811311 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.811213    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\") pod \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\" (UID: \"401d6f8c-5480-410d-99a4-f81f4ffb2d8b\") "
Oct 30 22:10:28.811947 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.811907    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca" pod "401d6f8c-5480-410d-99a4-f81f4ffb2d8b" (UID: "401d6f8c-5480-410d-99a4-f81f4ffb2d8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:28.813430 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.813404    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s" pod "401d6f8c-5480-410d-99a4-f81f4ffb2d8b" (UID: "401d6f8c-5480-410d-99a4-f81f4ffb2d8b"). InnerVolumeSpecName "kube-api-access-qhd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:28.814237 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.814214    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca" pod "401d6f8c-5480-410d-99a4-f81f4ffb2d8b" (UID: "401d6f8c-5480-410d-99a4-f81f4ffb2d8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:28.913158 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.913120    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:28.913158 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.913152    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:28.913158 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:28.913163    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhd6s\" (UniqueName: \"kubernetes.io/projected/401d6f8c-5480-410d-99a4-f81f4ffb2d8b-kube-api-access-qhd6s\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:29.308096 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:29.308060    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9" event={"ID":"401d6f8c-5480-410d-99a4-f81f4ffb2d8b","Type":"ContainerDied","Data":"dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65"}
Oct 30 22:10:29.308286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:29.308118    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc6354bd57b49031096684c59a434a390d6bfc427fa8d28d1473589554d26a65"
Oct 30 22:10:29.308286 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:29.308077    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"
Oct 30 22:10:33.672607 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:33.672570    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:10:36.369008 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:36.368962    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:10:37.918962 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:37.918913    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:42.520508 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.520451    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"]
Oct 30 22:10:42.521606 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.521575    7037 cpu_manager.go:414] "containerMap: removing container" podUID="401d6f8c-5480-410d-99a4-f81f4ffb2d8b" containerName="revoke-keys"
Oct 30 22:10:42.521606 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.521599    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401d6f8c-5480-410d-99a4-f81f4ffb2d8b" containerName="revoke-keys"
Oct 30 22:10:42.521776 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.521694    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="401d6f8c-5480-410d-99a4-f81f4ffb2d8b" containerName="revoke-keys"
Oct 30 22:10:42.524979 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.524962    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.550110 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.550083    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:42.550110 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.550092    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:42.550314 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.550092    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:10:42.550358 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.550332    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:42.552268 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.552245    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"]
Oct 30 22:10:42.686921 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.686881    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.687101 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.686928    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5kz\" (UniqueName: \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.687164 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.687106    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.788714 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.788624    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.788714 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.788670    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5kz\" (UniqueName: \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.788952 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.788784    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.789386 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.789356    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.791523 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.791479    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.807695 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.807668    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5kz\" (UniqueName: \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\") pod \"maas-api-revoke-sub-3e31a506-lp442\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:42.838454 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:42.838427    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:43.133126 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:43.133086    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"]
Oct 30 22:10:43.136674 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:10:43.136640    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81b9af7_2f76_48a3_9076_656ecd10cfb1.slice/crio-26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc WatchSource:0}: Error finding container 26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc: Status 404 returned error can't find the container with id 26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc
Oct 30 22:10:43.401243 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:43.401153    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442" event={"ID":"b81b9af7-2f76-48a3-9076-656ecd10cfb1","Type":"ContainerStarted","Data":"26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc"}
Oct 30 22:10:44.408445 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:44.408354    7037 generic.go:356] "Generic (PLEG): container finished" podID="b81b9af7-2f76-48a3-9076-656ecd10cfb1" containerID="fe54424f9e70461c59e4b833de62495473c4d2a0104bb477203a891676dbca83" exitCode=0
Oct 30 22:10:44.408855 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:44.408453    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442" event={"ID":"b81b9af7-2f76-48a3-9076-656ecd10cfb1","Type":"ContainerDied","Data":"fe54424f9e70461c59e4b833de62495473c4d2a0104bb477203a891676dbca83"}
Oct 30 22:10:45.812801 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.812772    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:45.923565 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.923526    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\") pod \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") "
Oct 30 22:10:45.923770 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.923632    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\") pod \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") "
Oct 30 22:10:45.923770 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.923706    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\" (UniqueName: \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\") pod \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\" (UID: \"b81b9af7-2f76-48a3-9076-656ecd10cfb1\") "
Oct 30 22:10:45.924036 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.924012    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca" pod "b81b9af7-2f76-48a3-9076-656ecd10cfb1" (UID: "b81b9af7-2f76-48a3-9076-656ecd10cfb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:45.925663 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.925639    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca" pod "b81b9af7-2f76-48a3-9076-656ecd10cfb1" (UID: "b81b9af7-2f76-48a3-9076-656ecd10cfb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:45.925774 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:45.925761    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz" pod "b81b9af7-2f76-48a3-9076-656ecd10cfb1" (UID: "b81b9af7-2f76-48a3-9076-656ecd10cfb1"). InnerVolumeSpecName "kube-api-access-dj5kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:46.024688 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.024607    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5kz\" (UniqueName: \"kubernetes.io/projected/b81b9af7-2f76-48a3-9076-656ecd10cfb1-kube-api-access-dj5kz\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:46.024688 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.024638    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:46.024688 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.024648    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81b9af7-2f76-48a3-9076-656ecd10cfb1-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:46.421594 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.421561    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"
Oct 30 22:10:46.421805 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.421599    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442" event={"ID":"b81b9af7-2f76-48a3-9076-656ecd10cfb1","Type":"ContainerDied","Data":"26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc"}
Oct 30 22:10:46.421805 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:46.421657    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26fe3ce3df4f5a398f47f07c3b8b8ec8de612ba1216f76c66973179efa27dbdc"
Oct 30 22:10:51.389652 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.389604    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" podUID="a4edddff-2d2b-4918-a86c-ceb91188699c" containerName="main" containerID="cri-o://ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611" gracePeriod=45
Oct 30 22:10:51.643881 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.643821    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:10:51.682177 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682146    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682227    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682330    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682465 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682356    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682465 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682405    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682603 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682485    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\" (UniqueName: \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\") pod \"a4edddff-2d2b-4918-a86c-ceb91188699c\" (UID: \"a4edddff-2d2b-4918-a86c-ceb91188699c\") "
Oct 30 22:10:51.682665 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682621    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:51.682719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682657    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:51.682719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682684    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:51.682945 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682930    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:51.683028 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682948    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:51.683028 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.682967    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:51.684538 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.684507    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:51.684664 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.684645    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:51.684775 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.684754    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh" pod "a4edddff-2d2b-4918-a86c-ceb91188699c" (UID: "a4edddff-2d2b-4918-a86c-ceb91188699c"). InnerVolumeSpecName "kube-api-access-7qfwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:51.783794 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.783757    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfwh\" (UniqueName: \"kubernetes.io/projected/a4edddff-2d2b-4918-a86c-ceb91188699c-kube-api-access-7qfwh\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:51.783794 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.783790    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4edddff-2d2b-4918-a86c-ceb91188699c-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:51.784063 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:51.783805    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4edddff-2d2b-4918-a86c-ceb91188699c-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:52.455679 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.455646    7037 generic.go:356] "Generic (PLEG): container finished" podID="a4edddff-2d2b-4918-a86c-ceb91188699c" containerID="ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611" exitCode=0
Oct 30 22:10:52.456148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.455692    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"
Oct 30 22:10:52.456148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.455708    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" event={"ID":"a4edddff-2d2b-4918-a86c-ceb91188699c","Type":"ContainerDied","Data":"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"}
Oct 30 22:10:52.456148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.455738    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz" event={"ID":"a4edddff-2d2b-4918-a86c-ceb91188699c","Type":"ContainerDied","Data":"f629d71f6fd3960f829b2df23ed41dd35738a1fb3f4fe91e6f703b97390adaea"}
Oct 30 22:10:52.456148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.455756    7037 scope.go:122] "RemoveContainer" containerID="ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"
Oct 30 22:10:52.465046 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.465026    7037 scope.go:122] "RemoveContainer" containerID="ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"
Oct 30 22:10:52.465312 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:52.465295    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611\": container with ID starting with ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611 not found: ID does not exist" containerID="ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"
Oct 30 22:10:52.465392 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.465324    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611"} err="failed to get container status \"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611\": rpc error: code = NotFound desc = could not find container \"ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611\": container with ID starting with ff945df8beba21f1caeaf1967cf0a4dec0ae87dad0c52b02374cdeb0e5cde611 not found: ID does not exist"
Oct 30 22:10:52.501924 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.501902    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:10:52.512695 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.512672    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-893c35ed-kserve-77df64f589-gkpkz"]
Oct 30 22:10:52.706591 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.706519    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4edddff-2d2b-4918-a86c-ceb91188699c" path="/var/lib/kubelet/pods/a4edddff-2d2b-4918-a86c-ceb91188699c/volumes"
Oct 30 22:10:52.940701 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:52.940667    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" podUID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" containerName="main" containerID="cri-o://83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460" gracePeriod=45
Oct 30 22:10:53.195667 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.195635    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:53.298828 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.298742    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.298828 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.298794    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\" (UniqueName: \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.299072 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.298875    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.299072 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.298920    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.299072 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299021    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.299239 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299082    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\") pod \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\" (UID: \"eca6b3ad-29d0-4c55-b808-c6f216a10f0b\") "
Oct 30 22:10:53.299239 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299100    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:53.299239 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299179    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:53.299372 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299269    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:53.299572 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299554    7037 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-model-cache\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.299641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299577    7037 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tmp-dir\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.299641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.299592    7037 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-home\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.300997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.300974    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "kube-api-access-c2knf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:53.301084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.301005    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:53.301084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.301065    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm" pod "eca6b3ad-29d0-4c55-b808-c6f216a10f0b" (UID: "eca6b3ad-29d0-4c55-b808-c6f216a10f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:53.400148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.400123    7037 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-dshm\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.400148 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.400147    7037 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-tls-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.400351 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.400158    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2knf\" (UniqueName: \"kubernetes.io/projected/eca6b3ad-29d0-4c55-b808-c6f216a10f0b-kube-api-access-c2knf\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:10:53.463233 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.463197    7037 generic.go:356] "Generic (PLEG): container finished" podID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" containerID="83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460" exitCode=0
Oct 30 22:10:53.463656 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.463250    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"
Oct 30 22:10:53.463656 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.463283    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" event={"ID":"eca6b3ad-29d0-4c55-b808-c6f216a10f0b","Type":"ContainerDied","Data":"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"}
Oct 30 22:10:53.463656 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.463317    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w" event={"ID":"eca6b3ad-29d0-4c55-b808-c6f216a10f0b","Type":"ContainerDied","Data":"e19d31953001527e3a27d6f8e32a38105f38f4f0486206e8ca1e6607586aa5a5"}
Oct 30 22:10:53.463656 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.463334    7037 scope.go:122] "RemoveContainer" containerID="83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"
Oct 30 22:10:53.473084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.473063    7037 scope.go:122] "RemoveContainer" containerID="83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"
Oct 30 22:10:53.473361 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:10:53.473342    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460\": container with ID starting with 83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460 not found: ID does not exist" containerID="83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"
Oct 30 22:10:53.473402 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.473371    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460"} err="failed to get container status \"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460\": rpc error: code = NotFound desc = could not find container \"83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460\": container with ID starting with 83717cda3855b0ab0e20ce4feae7385d5584b1831d32a5f7881bf4e2e2f5d460 not found: ID does not exist"
Oct 30 22:10:53.508602 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.508574    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:53.519184 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:53.519160    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-893c35ed-kserve-65b7958889-rd49w"]
Oct 30 22:10:54.708813 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:54.708782    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" path="/var/lib/kubelet/pods/eca6b3ad-29d0-4c55-b808-c6f216a10f0b/volumes"
Oct 30 22:10:59.658560 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:59.658526    7037 scope.go:122] "RemoveContainer" containerID="b99700050357ba89a209f9c1bd1e541b6e51af8f1605d9b6191988c3c4f135ba"
Oct 30 22:10:59.704029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:10:59.704003    7037 scope.go:122] "RemoveContainer" containerID="4d87851d0aa6d6bbdcdea246ee3350201e86d71c1af9c4768e2e156f074d5cd7"
Oct 30 22:11:03.672317 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:11:03.672280    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:11:05.983084 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983050    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"]
Oct 30 22:11:05.983629 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983610    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a4edddff-2d2b-4918-a86c-ceb91188699c" containerName="main"
Oct 30 22:11:05.983694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983631    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4edddff-2d2b-4918-a86c-ceb91188699c" containerName="main"
Oct 30 22:11:05.983694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983648    7037 cpu_manager.go:414] "containerMap: removing container" podUID="b81b9af7-2f76-48a3-9076-656ecd10cfb1" containerName="revoke-keys"
Oct 30 22:11:05.983694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983654    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81b9af7-2f76-48a3-9076-656ecd10cfb1" containerName="revoke-keys"
Oct 30 22:11:05.983694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983686    7037 cpu_manager.go:414] "containerMap: removing container" podUID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" containerName="main"
Oct 30 22:11:05.983694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983691    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" containerName="main"
Oct 30 22:11:05.983899 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983750    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca6b3ad-29d0-4c55-b808-c6f216a10f0b" containerName="main"
Oct 30 22:11:05.983899 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983759    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4edddff-2d2b-4918-a86c-ceb91188699c" containerName="main"
Oct 30 22:11:05.983899 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.983768    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81b9af7-2f76-48a3-9076-656ecd10cfb1" containerName="revoke-keys"
Oct 30 22:11:05.987537 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:05.987510    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.005409 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.005378    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"]
Oct 30 22:11:06.013838 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.013800    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:06.014014 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.013844    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:06.014014 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.013878    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\""
Oct 30 22:11:06.014014 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.013935    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:06.069588 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.069551    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.069768 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.069666    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.069768 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.069719    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqf2\" (UniqueName: \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.170933 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.170894    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.171128 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.170951    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqf2\" (UniqueName: \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.171128 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.171030    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.171694 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.171673    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.173227 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.173207    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.189397 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.189374    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqf2\" (UniqueName: \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\") pod \"maas-api-revoke-sub-c3439857-dtmxq\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.301410 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.301326    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:06.594849 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:06.594815    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"]
Oct 30 22:11:06.599480 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:06.599451    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72cb22d_0df4_4d2d_9f6e_9b1228205d9a.slice/crio-fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb WatchSource:0}: Error finding container fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb: Status 404 returned error can't find the container with id fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb
Oct 30 22:11:07.565501 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:07.565450    7037 generic.go:356] "Generic (PLEG): container finished" podID="a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" containerID="b336ac2f5154643540785366eda3598c8f6a6d09edd04d5097be7e103fd72f37" exitCode=0
Oct 30 22:11:07.565916 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:07.565537    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq" event={"ID":"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a","Type":"ContainerDied","Data":"b336ac2f5154643540785366eda3598c8f6a6d09edd04d5097be7e103fd72f37"}
Oct 30 22:11:07.565916 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:07.565590    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq" event={"ID":"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a","Type":"ContainerStarted","Data":"fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb"}
Oct 30 22:11:08.709394 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.709369    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:08.799005 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.798972    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\" (UniqueName: \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\") pod \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") "
Oct 30 22:11:08.799174 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.799022    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\") pod \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") "
Oct 30 22:11:08.799174 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.799124    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\") pod \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\" (UID: \"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a\") "
Oct 30 22:11:08.799390 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.799367    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca" pod "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" (UID: "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:08.801183 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.801158    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca" pod "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" (UID: "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:08.801183 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.801176    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2" pod "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" (UID: "a72cb22d-0df4-4d2d-9f6e-9b1228205d9a"). InnerVolumeSpecName "kube-api-access-sjqf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:08.900509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.900411    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqf2\" (UniqueName: \"kubernetes.io/projected/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-kube-api-access-sjqf2\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.900509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.900437    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-openshift-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.900509 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:08.900458    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a-maas-api-xks-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.576423 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:09.576393    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"
Oct 30 22:11:09.576641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:09.576429    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq" event={"ID":"a72cb22d-0df4-4d2d-9f6e-9b1228205d9a","Type":"ContainerDied","Data":"fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb"}
Oct 30 22:11:09.576641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:09.576479    7037 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe413b73c5af0745820dda707a02af6853b4f78575b4117c5c19f3a48acf0cb"
Oct 30 22:11:10.047366 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:10.047278    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"]
Oct 30 22:11:10.057593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:10.057560    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75pg"]
Oct 30 22:11:10.705195 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:10.705164    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb13615c-ea38-4dda-b3b4-a07983cfca36" path="/var/lib/kubelet/pods/eb13615c-ea38-4dda-b3b4-a07983cfca36/volumes"
Oct 30 22:11:18.358452 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.358408    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:18.358878 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.358864    7037 cpu_manager.go:414] "containerMap: removing container" podUID="a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" containerName="revoke-keys"
Oct 30 22:11:18.358927 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.358878    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" containerName="revoke-keys"
Oct 30 22:11:18.359080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.359068    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" containerName="revoke-keys"
Oct 30 22:11:18.363655 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.363633    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.388845 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.388821    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:18.389135 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.389109    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pdl4\""
Oct 30 22:11:18.389285 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.389258    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:11:18.389374 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.389334    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:18.391098 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.391073    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"]
Oct 30 22:11:18.393330 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.393303    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.393440 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.393353    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.393541 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.393507    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.393600 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.393563    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpfn\" (UniqueName: \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.395155 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.395139    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.399745 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.399731    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:11:18.402640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.402619    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:18.412774 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.412752    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"]
Oct 30 22:11:18.494343 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494303    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b602ddb0-2bbd-460e-814a-53cb756561e5-openshift-service-ca\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.494575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494359    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.494575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494463    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.494575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494527    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpfn\" (UniqueName: \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.494575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494553    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-metrics-certs\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.494575 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494575    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xkc\" (UniqueName: \"kubernetes.io/projected/b602ddb0-2bbd-460e-814a-53cb756561e5-kube-api-access-n8xkc\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.494900 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494710    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-maas-api-tls\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.494900 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.494775    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.495213 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.495192    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.496976 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.496949    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.497077 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.497060    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.512730 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.512701    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpfn\" (UniqueName: \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\") pod \"maas-api-5b898bf6b9-zh5ts\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") " pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.595542 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.595507    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-maas-api-tls\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.595721 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.595606    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b602ddb0-2bbd-460e-814a-53cb756561e5-openshift-service-ca\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.595721 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.595707    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-metrics-certs\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.595841 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.595727    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xkc\" (UniqueName: \"kubernetes.io/projected/b602ddb0-2bbd-460e-814a-53cb756561e5-kube-api-access-n8xkc\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.596295 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.596267    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b602ddb0-2bbd-460e-814a-53cb756561e5-openshift-service-ca\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.598034 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.598012    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-metrics-certs\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.598111 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.598056    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b602ddb0-2bbd-460e-814a-53cb756561e5-maas-api-tls\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.613964 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.613904    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xkc\" (UniqueName: \"kubernetes.io/projected/b602ddb0-2bbd-460e-814a-53cb756561e5-kube-api-access-n8xkc\") pod \"maas-api-54fbf74c89-g26bb\" (UID: \"b602ddb0-2bbd-460e-814a-53cb756561e5\") " pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.677771 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.677735    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:18.708926 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.708892    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:18.982100 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.982029    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:18.985386 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:18.985348    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04be9779_7322_4c7f_8bec_018f587bb13c.slice/crio-158c9f00ccd477528a6e11f341d59ec0041b32a21a0df77183569428234d786d WatchSource:0}: Error finding container 158c9f00ccd477528a6e11f341d59ec0041b32a21a0df77183569428234d786d: Status 404 returned error can't find the container with id 158c9f00ccd477528a6e11f341d59ec0041b32a21a0df77183569428234d786d
Oct 30 22:11:18.986787 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:18.986770    7037 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:19.037354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.037314    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"]
Oct 30 22:11:19.041441 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:19.041407    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb602ddb0_2bbd_460e_814a_53cb756561e5.slice/crio-543ab12f12c84d36e0596bec4a371d365021089bec03b30f809385ce6df099f8 WatchSource:0}: Error finding container 543ab12f12c84d36e0596bec4a371d365021089bec03b30f809385ce6df099f8: Status 404 returned error can't find the container with id 543ab12f12c84d36e0596bec4a371d365021089bec03b30f809385ce6df099f8
Oct 30 22:11:19.626318 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.626218    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" event={"ID":"04be9779-7322-4c7f-8bec-018f587bb13c","Type":"ContainerStarted","Data":"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"}
Oct 30 22:11:19.626318 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.626293    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" event={"ID":"04be9779-7322-4c7f-8bec-018f587bb13c","Type":"ContainerStarted","Data":"158c9f00ccd477528a6e11f341d59ec0041b32a21a0df77183569428234d786d"}
Oct 30 22:11:19.626847 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.626452    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:19.627981 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.627943    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb" event={"ID":"b602ddb0-2bbd-460e-814a-53cb756561e5","Type":"ContainerStarted","Data":"b47700be27acfb84f9f4a7454f06101c2620ec132a1aa6a4166e77e70703e225"}
Oct 30 22:11:19.628129 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.627998    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb" event={"ID":"b602ddb0-2bbd-460e-814a-53cb756561e5","Type":"ContainerStarted","Data":"543ab12f12c84d36e0596bec4a371d365021089bec03b30f809385ce6df099f8"}
Oct 30 22:11:19.628129 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.628031    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:19.665993 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.665940    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" podStartSLOduration=1.332543632 podStartE2EDuration="1.665924671s" totalImagesPullingTime="333.381039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:19.664602555 +0000 UTC m=+4223.076142005" watchObservedRunningTime="2026-10-30 22:11:19.665924671 +0000 UTC m=+4223.077464142"
Oct 30 22:11:19.702447 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:19.702382    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb" podStartSLOduration=1.381139383 podStartE2EDuration="1.702366116s" totalImagesPullingTime="321.226733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:19.701667711 +0000 UTC m=+4223.113207151" watchObservedRunningTime="2026-10-30 22:11:19.702366116 +0000 UTC m=+4223.113905553"
Oct 30 22:11:25.639243 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:25.639206    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:25.639728 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:25.639681    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54fbf74c89-g26bb"
Oct 30 22:11:25.793630 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:25.793577    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:25.793838 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:25.793804    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" podUID="04be9779-7322-4c7f-8bec-018f587bb13c" containerName="maas-api" containerID="cri-o://081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657" gracePeriod=30
Oct 30 22:11:26.056449 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.056422    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:26.174867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.174772    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\") pod \"04be9779-7322-4c7f-8bec-018f587bb13c\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") "
Oct 30 22:11:26.174867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.174830    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\") pod \"04be9779-7322-4c7f-8bec-018f587bb13c\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") "
Oct 30 22:11:26.174867 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.174859    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\" (UniqueName: \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\") pod \"04be9779-7322-4c7f-8bec-018f587bb13c\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") "
Oct 30 22:11:26.175166 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.174916    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\") pod \"04be9779-7322-4c7f-8bec-018f587bb13c\" (UID: \"04be9779-7322-4c7f-8bec-018f587bb13c\") "
Oct 30 22:11:26.175235 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.175193    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca" pod "04be9779-7322-4c7f-8bec-018f587bb13c" (UID: "04be9779-7322-4c7f-8bec-018f587bb13c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:26.175316 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.175292    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04be9779-7322-4c7f-8bec-018f587bb13c-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:26.176997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.176969    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls" pod "04be9779-7322-4c7f-8bec-018f587bb13c" (UID: "04be9779-7322-4c7f-8bec-018f587bb13c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:26.177154 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.177132    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs" pod "04be9779-7322-4c7f-8bec-018f587bb13c" (UID: "04be9779-7322-4c7f-8bec-018f587bb13c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:26.177226 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.177164    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn" pod "04be9779-7322-4c7f-8bec-018f587bb13c" (UID: "04be9779-7322-4c7f-8bec-018f587bb13c"). InnerVolumeSpecName "kube-api-access-hdpfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:26.276405 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.276369    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:26.276405 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.276401    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdpfn\" (UniqueName: \"kubernetes.io/projected/04be9779-7322-4c7f-8bec-018f587bb13c-kube-api-access-hdpfn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:26.276405 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.276411    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04be9779-7322-4c7f-8bec-018f587bb13c-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:26.664267 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.664231    7037 generic.go:356] "Generic (PLEG): container finished" podID="04be9779-7322-4c7f-8bec-018f587bb13c" containerID="081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657" exitCode=0
Oct 30 22:11:26.664726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.664279    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"
Oct 30 22:11:26.664726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.664312    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" event={"ID":"04be9779-7322-4c7f-8bec-018f587bb13c","Type":"ContainerDied","Data":"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"}
Oct 30 22:11:26.664726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.664368    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts" event={"ID":"04be9779-7322-4c7f-8bec-018f587bb13c","Type":"ContainerDied","Data":"158c9f00ccd477528a6e11f341d59ec0041b32a21a0df77183569428234d786d"}
Oct 30 22:11:26.664726 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.664389    7037 scope.go:122] "RemoveContainer" containerID="081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"
Oct 30 22:11:26.674752 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.674730    7037 scope.go:122] "RemoveContainer" containerID="081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"
Oct 30 22:11:26.675025 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:11:26.675006    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657\": container with ID starting with 081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657 not found: ID does not exist" containerID="081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"
Oct 30 22:11:26.675067 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.675035    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657"} err="failed to get container status \"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657\": rpc error: code = NotFound desc = could not find container \"081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657\": container with ID starting with 081b11ccdcd61bb64598135c039d427040fbee06aa5c9b37064f5538bc9ac657 not found: ID does not exist"
Oct 30 22:11:26.720061 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.720029    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:26.726983 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:26.726956    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b898bf6b9-zh5ts"]
Oct 30 22:11:28.706160 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:28.706124    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04be9779-7322-4c7f-8bec-018f587bb13c" path="/var/lib/kubelet/pods/04be9779-7322-4c7f-8bec-018f587bb13c/volumes"
Oct 30 22:11:31.990121 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:31.990084    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:31.990651 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:31.990633    7037 cpu_manager.go:414] "containerMap: removing container" podUID="04be9779-7322-4c7f-8bec-018f587bb13c" containerName="maas-api"
Oct 30 22:11:31.990718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:31.990652    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04be9779-7322-4c7f-8bec-018f587bb13c" containerName="maas-api"
Oct 30 22:11:31.990820 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:31.990761    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="04be9779-7322-4c7f-8bec-018f587bb13c" containerName="maas-api"
Oct 30 22:11:31.995360 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:31.995343    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.014371 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.014340    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6ad3224d-serving-cert\""
Oct 30 22:11:32.026956 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.026928    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:32.049424 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.049397    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:11:32.053327 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.053308    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.068732 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.068711    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6ad3224d\""
Oct 30 22:11:32.073301 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.073272    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:11:32.083314 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.083215    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6ad3224d-dockercfg-bb9cj\""
Oct 30 22:11:32.124369 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.124339    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:11:32.128473 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.128452    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.132273 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132247    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.132362 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132319    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tjn\" (UniqueName: \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.132413 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132385    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65w5\" (UniqueName: \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.132451 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132429    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.132542 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132519    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.132676 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.132638    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.145873 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.145849    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:11:32.233354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233314    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvln\" (UniqueName: \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.233354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233359    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.233620 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233480    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.233620 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233563    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.233697 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233629    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tjn\" (UniqueName: \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.233697 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233655    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.233776 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233697    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65w5\" (UniqueName: \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.233827 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.233800    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.234332 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.234305    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.234465 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.234312    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.235940 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.235916    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.236141 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.236123    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.251573 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.251519    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tjn\" (UniqueName: \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\") pod \"payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.251747 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.251729    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65w5\" (UniqueName: \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\") pod \"maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.308886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.308858    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:32.334981 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.334949    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.335141 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.335033    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvln\" (UniqueName: \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.335724 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.335699    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.353127 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.353101    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvln\" (UniqueName: \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\") pod \"payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") " pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.372603 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.370887    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:32.444007 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.443104    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:32.605698 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.605663    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:32.609261 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:32.609234    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b77a3ee_503d_4551_a6e6_bbdef94e66d7.slice/crio-571e4f49bd0d966c4142bc5a2dfa0d51a5bb14a226767a919d40820dcc83299a WatchSource:0}: Error finding container 571e4f49bd0d966c4142bc5a2dfa0d51a5bb14a226767a919d40820dcc83299a: Status 404 returned error can't find the container with id 571e4f49bd0d966c4142bc5a2dfa0d51a5bb14a226767a919d40820dcc83299a
Oct 30 22:11:32.687064 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.687033    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:11:32.690330 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:32.690298    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f05f4e4_c344_484b_b232_99bce1879310.slice/crio-f6c45071b15c4d64b414bac8e11f3988fdfbbafb009b4bf1c6e48ea694b5646e WatchSource:0}: Error finding container f6c45071b15c4d64b414bac8e11f3988fdfbbafb009b4bf1c6e48ea694b5646e: Status 404 returned error can't find the container with id f6c45071b15c4d64b414bac8e11f3988fdfbbafb009b4bf1c6e48ea694b5646e
Oct 30 22:11:32.695658 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.695631    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" event={"ID":"6b77a3ee-503d-4551-a6e6-bbdef94e66d7","Type":"ContainerStarted","Data":"571e4f49bd0d966c4142bc5a2dfa0d51a5bb14a226767a919d40820dcc83299a"}
Oct 30 22:11:32.762787 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:32.762722    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:11:32.766416 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:11:32.766380    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c462ad_aab0_4914_b1ac_aecc58f17f92.slice/crio-04ccebc787600adb7ad03da2702c4ae1495ef652a1052266e32adb4a674b5722 WatchSource:0}: Error finding container 04ccebc787600adb7ad03da2702c4ae1495ef652a1052266e32adb4a674b5722: Status 404 returned error can't find the container with id 04ccebc787600adb7ad03da2702c4ae1495ef652a1052266e32adb4a674b5722
Oct 30 22:11:33.674565 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:11:33.674520    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:11:33.702764 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.702726    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" event={"ID":"6b77a3ee-503d-4551-a6e6-bbdef94e66d7","Type":"ContainerStarted","Data":"d5172a83e9a80b143fd0060b91c9c3014687979ffa1b80bacceeb747a36db295"}
Oct 30 22:11:33.702933 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.702847    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:33.704546 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.704519    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" event={"ID":"33c462ad-aab0-4914-b1ac-aecc58f17f92","Type":"ContainerStarted","Data":"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"}
Oct 30 22:11:33.704668 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.704565    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" event={"ID":"33c462ad-aab0-4914-b1ac-aecc58f17f92","Type":"ContainerStarted","Data":"04ccebc787600adb7ad03da2702c4ae1495ef652a1052266e32adb4a674b5722"}
Oct 30 22:11:33.704668 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.704595    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:33.706124 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.706099    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" event={"ID":"8f05f4e4-c344-484b-b232-99bce1879310","Type":"ContainerStarted","Data":"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"}
Oct 30 22:11:33.706229 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.706150    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:33.706229 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.706174    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" event={"ID":"8f05f4e4-c344-484b-b232-99bce1879310","Type":"ContainerStarted","Data":"f6c45071b15c4d64b414bac8e11f3988fdfbbafb009b4bf1c6e48ea694b5646e"}
Oct 30 22:11:33.747627 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.747576    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" podStartSLOduration=2.472006614 podStartE2EDuration="2.747558514s" totalImagesPullingTime="275.5519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:33.745597018 +0000 UTC m=+4237.157136457" watchObservedRunningTime="2026-10-30 22:11:33.747558514 +0000 UTC m=+4237.159097952"
Oct 30 22:11:33.783136 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.783095    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" podStartSLOduration=1.7830812969999998 podStartE2EDuration="1.783081297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:33.78220748 +0000 UTC m=+4237.193746918" watchObservedRunningTime="2026-10-30 22:11:33.783081297 +0000 UTC m=+4237.194620734"
Oct 30 22:11:33.819533 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:33.819465    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" podStartSLOduration=1.819451655 podStartE2EDuration="1.819451655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:33.817317433 +0000 UTC m=+4237.228856873" watchObservedRunningTime="2026-10-30 22:11:33.819451655 +0000 UTC m=+4237.230991091"
Oct 30 22:11:39.717131 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:39.717056    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:44.714578 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:44.714548    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:11:44.715121 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:44.714627    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:11:55.311751 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:55.311717    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:11:55.343419 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:55.343385    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:11:56.614698 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.614664    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:56.615170 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.614871    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" podUID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" containerName="maas-api" containerID="cri-o://d5172a83e9a80b143fd0060b91c9c3014687979ffa1b80bacceeb747a36db295" gracePeriod=30
Oct 30 22:11:56.835080 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.835038    7037 generic.go:356] "Generic (PLEG): container finished" podID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" containerID="d5172a83e9a80b143fd0060b91c9c3014687979ffa1b80bacceeb747a36db295" exitCode=0
Oct 30 22:11:56.835348 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.835132    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" event={"ID":"6b77a3ee-503d-4551-a6e6-bbdef94e66d7","Type":"ContainerDied","Data":"d5172a83e9a80b143fd0060b91c9c3014687979ffa1b80bacceeb747a36db295"}
Oct 30 22:11:56.885702 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.885675    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:56.984388 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.984353    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\" (UniqueName: \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\") pod \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") "
Oct 30 22:11:56.984388 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.984397    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\") pod \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") "
Oct 30 22:11:56.984705 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.984461    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\") pod \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") "
Oct 30 22:11:56.984705 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.984519    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\") pod \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\" (UID: \"6b77a3ee-503d-4551-a6e6-bbdef94e66d7\") "
Oct 30 22:11:56.984909 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.984879    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca" pod "6b77a3ee-503d-4551-a6e6-bbdef94e66d7" (UID: "6b77a3ee-503d-4551-a6e6-bbdef94e66d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:56.986682 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.986653    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls" pod "6b77a3ee-503d-4551-a6e6-bbdef94e66d7" (UID: "6b77a3ee-503d-4551-a6e6-bbdef94e66d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:56.986796 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.986680    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs" pod "6b77a3ee-503d-4551-a6e6-bbdef94e66d7" (UID: "6b77a3ee-503d-4551-a6e6-bbdef94e66d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:56.986796 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:56.986737    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5" pod "6b77a3ee-503d-4551-a6e6-bbdef94e66d7" (UID: "6b77a3ee-503d-4551-a6e6-bbdef94e66d7"). InnerVolumeSpecName "kube-api-access-d65w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:57.085847 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.085815    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65w5\" (UniqueName: \"kubernetes.io/projected/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-kube-api-access-d65w5\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:57.085847 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.085842    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:57.085847 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.085852    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:57.086101 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.085860    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b77a3ee-503d-4551-a6e6-bbdef94e66d7-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:11:57.840902 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.840866    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"
Oct 30 22:11:57.840902 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.840879    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm" event={"ID":"6b77a3ee-503d-4551-a6e6-bbdef94e66d7","Type":"ContainerDied","Data":"571e4f49bd0d966c4142bc5a2dfa0d51a5bb14a226767a919d40820dcc83299a"}
Oct 30 22:11:57.841410 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.840934    7037 scope.go:122] "RemoveContainer" containerID="d5172a83e9a80b143fd0060b91c9c3014687979ffa1b80bacceeb747a36db295"
Oct 30 22:11:57.888013 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.887978    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:57.899054 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:57.899028    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ad3224d-b75cbc64f-blnfm"]
Oct 30 22:11:58.706896 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:11:58.706858    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" path="/var/lib/kubelet/pods/6b77a3ee-503d-4551-a6e6-bbdef94e66d7/volumes"
Oct 30 22:12:00.312343 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.312296    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" podUID="33c462ad-aab0-4914-b1ac-aecc58f17f92" containerName="payload-processing" containerID="cri-o://7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf" gracePeriod=25
Oct 30 22:12:00.343892 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.343820    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" podUID="8f05f4e4-c344-484b-b232-99bce1879310" containerName="payload-pre-processing" containerID="cri-o://26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a" gracePeriod=25
Oct 30 22:12:00.605083 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.605057    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:12:00.631260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.631235    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:12:00.722852 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.722821    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\" (UniqueName: \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\") pod \"33c462ad-aab0-4914-b1ac-aecc58f17f92\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") "
Oct 30 22:12:00.723038 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.722869    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\") pod \"33c462ad-aab0-4914-b1ac-aecc58f17f92\" (UID: \"33c462ad-aab0-4914-b1ac-aecc58f17f92\") "
Oct 30 22:12:00.723251 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.723229    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume" pod "33c462ad-aab0-4914-b1ac-aecc58f17f92" (UID: "33c462ad-aab0-4914-b1ac-aecc58f17f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:00.724886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.724855    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln" pod "33c462ad-aab0-4914-b1ac-aecc58f17f92" (UID: "33c462ad-aab0-4914-b1ac-aecc58f17f92"). InnerVolumeSpecName "kube-api-access-zzvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:00.824188 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.824158    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\") pod \"8f05f4e4-c344-484b-b232-99bce1879310\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") "
Oct 30 22:12:00.824395 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.824220    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\" (UniqueName: \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\") pod \"8f05f4e4-c344-484b-b232-99bce1879310\" (UID: \"8f05f4e4-c344-484b-b232-99bce1879310\") "
Oct 30 22:12:00.824650 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.824590    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume" pod "8f05f4e4-c344-484b-b232-99bce1879310" (UID: "8f05f4e4-c344-484b-b232-99bce1879310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:00.824650 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.824615    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzvln\" (UniqueName: \"kubernetes.io/projected/33c462ad-aab0-4914-b1ac-aecc58f17f92-kube-api-access-zzvln\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:12:00.824650 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.824638    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33c462ad-aab0-4914-b1ac-aecc58f17f92-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:12:00.826475 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.826448    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn" pod "8f05f4e4-c344-484b-b232-99bce1879310" (UID: "8f05f4e4-c344-484b-b232-99bce1879310"). InnerVolumeSpecName "kube-api-access-c7tjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:00.866011 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.865925    7037 generic.go:356] "Generic (PLEG): container finished" podID="33c462ad-aab0-4914-b1ac-aecc58f17f92" containerID="7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf" exitCode=0
Oct 30 22:12:00.866011 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.865974    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"
Oct 30 22:12:00.866260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.866017    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" event={"ID":"33c462ad-aab0-4914-b1ac-aecc58f17f92","Type":"ContainerDied","Data":"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"}
Oct 30 22:12:00.866260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.866082    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2" event={"ID":"33c462ad-aab0-4914-b1ac-aecc58f17f92","Type":"ContainerDied","Data":"04ccebc787600adb7ad03da2702c4ae1495ef652a1052266e32adb4a674b5722"}
Oct 30 22:12:00.866260 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.866111    7037 scope.go:122] "RemoveContainer" containerID="7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"
Oct 30 22:12:00.867538 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.867479    7037 generic.go:356] "Generic (PLEG): container finished" podID="8f05f4e4-c344-484b-b232-99bce1879310" containerID="26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a" exitCode=0
Oct 30 22:12:00.867641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.867526    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" event={"ID":"8f05f4e4-c344-484b-b232-99bce1879310","Type":"ContainerDied","Data":"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"}
Oct 30 22:12:00.867641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.867545    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"
Oct 30 22:12:00.867641 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.867579    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b" event={"ID":"8f05f4e4-c344-484b-b232-99bce1879310","Type":"ContainerDied","Data":"f6c45071b15c4d64b414bac8e11f3988fdfbbafb009b4bf1c6e48ea694b5646e"}
Oct 30 22:12:00.875657 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.875587    7037 scope.go:122] "RemoveContainer" containerID="7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"
Oct 30 22:12:00.875924 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:12:00.875896    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf\": container with ID starting with 7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf not found: ID does not exist" containerID="7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"
Oct 30 22:12:00.876009 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.875933    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf"} err="failed to get container status \"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf\": rpc error: code = NotFound desc = could not find container \"7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf\": container with ID starting with 7da360215c68aa95fdd8c8865f646f7539420bad2951e43580abea92a3be81bf not found: ID does not exist"
Oct 30 22:12:00.876009 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.875952    7037 scope.go:122] "RemoveContainer" containerID="26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"
Oct 30 22:12:00.884765 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.884744    7037 scope.go:122] "RemoveContainer" containerID="26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"
Oct 30 22:12:00.885041 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:12:00.885020    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a\": container with ID starting with 26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a not found: ID does not exist" containerID="26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"
Oct 30 22:12:00.885100 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.885049    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a"} err="failed to get container status \"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a\": rpc error: code = NotFound desc = could not find container \"26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a\": container with ID starting with 26cd77b776f808cc3a9376b8836217e0429ff6d28d6adf45e8fd6f037b4bf97a not found: ID does not exist"
Oct 30 22:12:00.911478 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.911446    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:12:00.914484 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.914458    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ad3224d-57fbf7b9bc-5tf9b"]
Oct 30 22:12:00.925434 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.925404    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tjn\" (UniqueName: \"kubernetes.io/projected/8f05f4e4-c344-484b-b232-99bce1879310-kube-api-access-c7tjn\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:12:00.925434 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.925427    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f05f4e4-c344-484b-b232-99bce1879310-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:12:00.944789 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.944759    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:12:00.948844 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:00.948812    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ad3224d-79cfdfb994-rnnj2"]
Oct 30 22:12:02.706047 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:02.706014    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c462ad-aab0-4914-b1ac-aecc58f17f92" path="/var/lib/kubelet/pods/33c462ad-aab0-4914-b1ac-aecc58f17f92/volumes"
Oct 30 22:12:02.706432 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:02.706408    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f05f4e4-c344-484b-b232-99bce1879310" path="/var/lib/kubelet/pods/8f05f4e4-c344-484b-b232-99bce1879310/volumes"
Oct 30 22:12:03.671403 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:12:03.671367    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:12:08.361778 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.361737    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:12:08.362477 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362456    7037 cpu_manager.go:414] "containerMap: removing container" podUID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" containerName="maas-api"
Oct 30 22:12:08.362640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362479    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" containerName="maas-api"
Oct 30 22:12:08.362640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362536    7037 cpu_manager.go:414] "containerMap: removing container" podUID="33c462ad-aab0-4914-b1ac-aecc58f17f92" containerName="payload-processing"
Oct 30 22:12:08.362640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362545    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c462ad-aab0-4914-b1ac-aecc58f17f92" containerName="payload-processing"
Oct 30 22:12:08.362640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362574    7037 cpu_manager.go:414] "containerMap: removing container" podUID="8f05f4e4-c344-484b-b232-99bce1879310" containerName="payload-pre-processing"
Oct 30 22:12:08.362640 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362582    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f05f4e4-c344-484b-b232-99bce1879310" containerName="payload-pre-processing"
Oct 30 22:12:08.362997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362708    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b77a3ee-503d-4551-a6e6-bbdef94e66d7" containerName="maas-api"
Oct 30 22:12:08.362997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362728    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c462ad-aab0-4914-b1ac-aecc58f17f92" containerName="payload-processing"
Oct 30 22:12:08.362997 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.362741    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f05f4e4-c344-484b-b232-99bce1879310" containerName="payload-pre-processing"
Oct 30 22:12:08.366022 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.366002    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.390263 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.390231    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.390445 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.390352    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86ls\" (UniqueName: \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.390445 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.390403    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.390445 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.390428    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.391962 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.391935    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8c3a04-serving-cert\""
Oct 30 22:12:08.393857 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.393836    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:12:08.491426 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.491394    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86ls\" (UniqueName: \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.491426 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.491439    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.491724 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.491465    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.491724 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.491584    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.492256 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.492227    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.494011 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.493983    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.494108 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.494038    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.516503 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.516470    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86ls\" (UniqueName: \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\") pod \"maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.679815 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.679727    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:08.761209 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.760471    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:12:08.765274 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.765246    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.779382 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.779357    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8c3a04\""
Oct 30 22:12:08.794649 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.794588    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.795673 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.794724    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66nl\" (UniqueName: \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.795848 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.795825    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:12:08.895590 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.895548    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.895802 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.895619    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66nl\" (UniqueName: \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.896408 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.896384    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.915068 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.915027    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66nl\" (UniqueName: \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\") pod \"payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:08.975020 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:08.974944    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:12:08.981211 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:12:08.980011    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb37aea_8239_4308_8cba_65f378bc439e.slice/crio-37249f7d485889ba2ccec929957087da9f5bb04e50177e405bc241deeea7d7f8 WatchSource:0}: Error finding container 37249f7d485889ba2ccec929957087da9f5bb04e50177e405bc241deeea7d7f8: Status 404 returned error can't find the container with id 37249f7d485889ba2ccec929957087da9f5bb04e50177e405bc241deeea7d7f8
Oct 30 22:12:09.080742 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.080705    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:09.377264 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.377235    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:12:09.392705 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:12:09.392669    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781cb114_f900_4962_97a3_9d185e6dad3e.slice/crio-505c12f022364d52a9a76a0f7a26419a034c8efd950c542aee43b431cfc04d0f WatchSource:0}: Error finding container 505c12f022364d52a9a76a0f7a26419a034c8efd950c542aee43b431cfc04d0f: Status 404 returned error can't find the container with id 505c12f022364d52a9a76a0f7a26419a034c8efd950c542aee43b431cfc04d0f
Oct 30 22:12:09.926183 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.926139    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" event={"ID":"781cb114-f900-4962-97a3-9d185e6dad3e","Type":"ContainerStarted","Data":"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"}
Oct 30 22:12:09.926561 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.926199    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" event={"ID":"781cb114-f900-4962-97a3-9d185e6dad3e","Type":"ContainerStarted","Data":"505c12f022364d52a9a76a0f7a26419a034c8efd950c542aee43b431cfc04d0f"}
Oct 30 22:12:09.926561 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.926230    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:09.927828 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.927804    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" event={"ID":"4bb37aea-8239-4308-8cba-65f378bc439e","Type":"ContainerStarted","Data":"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"}
Oct 30 22:12:09.927929 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.927846    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" event={"ID":"4bb37aea-8239-4308-8cba-65f378bc439e","Type":"ContainerStarted","Data":"37249f7d485889ba2ccec929957087da9f5bb04e50177e405bc241deeea7d7f8"}
Oct 30 22:12:09.927929 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.927871    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:09.963305 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:09.961898    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" podStartSLOduration=1.961880957 podStartE2EDuration="1.961880957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:09.960644033 +0000 UTC m=+4273.372183471" watchObservedRunningTime="2026-10-30 22:12:09.961880957 +0000 UTC m=+4273.373420403"
Oct 30 22:12:10.000317 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:10.000263    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" podStartSLOduration=1.625569399 podStartE2EDuration="2.000248742s" totalImagesPullingTime="374.679343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:09.998880931 +0000 UTC m=+4273.410420604" watchObservedRunningTime="2026-10-30 22:12:10.000248742 +0000 UTC m=+4273.411788179"
Oct 30 22:12:15.947132 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:15.947096    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:12:20.945660 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:20.945625    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:12:33.670836 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:12:33.670796    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:12:56.027299 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:56.027260    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"]
Oct 30 22:12:56.038123 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:56.038089    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8zmpf"]
Oct 30 22:12:56.707338 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:56.707307    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e970f63-426f-4540-9f73-1ed3325c99e2" path="/var/lib/kubelet/pods/2e970f63-426f-4540-9f73-1ed3325c99e2/volumes"
Oct 30 22:12:59.777449 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:12:59.777413    7037 scope.go:122] "RemoveContainer" containerID="c36fcf4092c413fa190f6190501005679f0af33cf21f16d8c880a0fe8251379c"
Oct 30 22:13:03.672258 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:13:03.672217    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:13:17.028362 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:17.028325    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"]
Oct 30 22:13:17.038834 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:17.038806    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zm6c4"]
Oct 30 22:13:18.707471 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:18.707440    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2328a09c-e579-41f5-aa43-a636a0ec0cb7" path="/var/lib/kubelet/pods/2328a09c-e579-41f5-aa43-a636a0ec0cb7/volumes"
Oct 30 22:13:33.671635 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:13:33.671596    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:13:52.641060 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:52.641026    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:13:57.641358 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:57.641295    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" podUID="781cb114-f900-4962-97a3-9d185e6dad3e" containerName="payload-processing" containerID="cri-o://c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad" gracePeriod=25
Oct 30 22:13:57.898156 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:57.898102    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:13:58.010045 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.010015    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\" (UniqueName: \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\") pod \"781cb114-f900-4962-97a3-9d185e6dad3e\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") "
Oct 30 22:13:58.010232 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.010176    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\") pod \"781cb114-f900-4962-97a3-9d185e6dad3e\" (UID: \"781cb114-f900-4962-97a3-9d185e6dad3e\") "
Oct 30 22:13:58.010604 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.010580    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume" pod "781cb114-f900-4962-97a3-9d185e6dad3e" (UID: "781cb114-f900-4962-97a3-9d185e6dad3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:58.012198 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.012178    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl" pod "781cb114-f900-4962-97a3-9d185e6dad3e" (UID: "781cb114-f900-4962-97a3-9d185e6dad3e"). InnerVolumeSpecName "kube-api-access-b66nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:58.111043 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.111009    7037 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/781cb114-f900-4962-97a3-9d185e6dad3e-plugins-config-volume\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.111043 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.111036    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b66nl\" (UniqueName: \"kubernetes.io/projected/781cb114-f900-4962-97a3-9d185e6dad3e-kube-api-access-b66nl\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.443173 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.443137    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:13:58.443424 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.443365    7037 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" podUID="4bb37aea-8239-4308-8cba-65f378bc439e" containerName="maas-api" containerID="cri-o://a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341" gracePeriod=30
Oct 30 22:13:58.507336 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.507293    7037 generic.go:356] "Generic (PLEG): container finished" podID="781cb114-f900-4962-97a3-9d185e6dad3e" containerID="c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad" exitCode=0
Oct 30 22:13:58.507566 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.507350    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"
Oct 30 22:13:58.507566 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.507346    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" event={"ID":"781cb114-f900-4962-97a3-9d185e6dad3e","Type":"ContainerDied","Data":"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"}
Oct 30 22:13:58.507566 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.507483    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn" event={"ID":"781cb114-f900-4962-97a3-9d185e6dad3e","Type":"ContainerDied","Data":"505c12f022364d52a9a76a0f7a26419a034c8efd950c542aee43b431cfc04d0f"}
Oct 30 22:13:58.507566 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.507537    7037 scope.go:122] "RemoveContainer" containerID="c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"
Oct 30 22:13:58.555151 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.555130    7037 scope.go:122] "RemoveContainer" containerID="c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"
Oct 30 22:13:58.555549 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:13:58.555519    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad\": container with ID starting with c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad not found: ID does not exist" containerID="c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"
Oct 30 22:13:58.555636 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.555560    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad"} err="failed to get container status \"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad\": rpc error: code = NotFound desc = could not find container \"c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad\": container with ID starting with c029f446454c5de56e12427f0768e923052d196bb4614dda87e83a84bee9faad not found: ID does not exist"
Oct 30 22:13:58.562415 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.562388    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:13:58.568771 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.568745    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8c3a04-594499f4c4-skfxn"]
Oct 30 22:13:58.703359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.703295    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:13:58.706046 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.706022    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781cb114-f900-4962-97a3-9d185e6dad3e" path="/var/lib/kubelet/pods/781cb114-f900-4962-97a3-9d185e6dad3e/volumes"
Oct 30 22:13:58.818486 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.818453    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\") pod \"4bb37aea-8239-4308-8cba-65f378bc439e\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") "
Oct 30 22:13:58.818676 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.818516    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\" (UniqueName: \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\") pod \"4bb37aea-8239-4308-8cba-65f378bc439e\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") "
Oct 30 22:13:58.818676 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.818550    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\") pod \"4bb37aea-8239-4308-8cba-65f378bc439e\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") "
Oct 30 22:13:58.818676 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.818596    7037 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\") pod \"4bb37aea-8239-4308-8cba-65f378bc439e\" (UID: \"4bb37aea-8239-4308-8cba-65f378bc439e\") "
Oct 30 22:13:58.818874 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.818852    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca" pod "4bb37aea-8239-4308-8cba-65f378bc439e" (UID: "4bb37aea-8239-4308-8cba-65f378bc439e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:58.819029 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.819006    7037 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb37aea-8239-4308-8cba-65f378bc439e-openshift-service-ca\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.820813 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.820782    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls" pod "4bb37aea-8239-4308-8cba-65f378bc439e" (UID: "4bb37aea-8239-4308-8cba-65f378bc439e"). InnerVolumeSpecName "kube-api-access-c86ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:58.820910 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.820813    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs" pod "4bb37aea-8239-4308-8cba-65f378bc439e" (UID: "4bb37aea-8239-4308-8cba-65f378bc439e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:58.820910 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.820825    7037 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls" pod "4bb37aea-8239-4308-8cba-65f378bc439e" (UID: "4bb37aea-8239-4308-8cba-65f378bc439e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:58.919813 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.919780    7037 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-maas-api-tls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.919813 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.919812    7037 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86ls\" (UniqueName: \"kubernetes.io/projected/4bb37aea-8239-4308-8cba-65f378bc439e-kube-api-access-c86ls\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:58.920042 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:58.919823    7037 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb37aea-8239-4308-8cba-65f378bc439e-metrics-certs\") on node \"ip-10-0-137-72.ec2.internal\" DevicePath \"\""
Oct 30 22:13:59.513832 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.513799    7037 generic.go:356] "Generic (PLEG): container finished" podID="4bb37aea-8239-4308-8cba-65f378bc439e" containerID="a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341" exitCode=0
Oct 30 22:13:59.514034 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.513854    7037 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"
Oct 30 22:13:59.514034 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.513885    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" event={"ID":"4bb37aea-8239-4308-8cba-65f378bc439e","Type":"ContainerDied","Data":"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"}
Oct 30 22:13:59.514034 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.513941    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf" event={"ID":"4bb37aea-8239-4308-8cba-65f378bc439e","Type":"ContainerDied","Data":"37249f7d485889ba2ccec929957087da9f5bb04e50177e405bc241deeea7d7f8"}
Oct 30 22:13:59.514034 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.513972    7037 scope.go:122] "RemoveContainer" containerID="a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"
Oct 30 22:13:59.523198 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.523178    7037 scope.go:122] "RemoveContainer" containerID="a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"
Oct 30 22:13:59.523459 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:13:59.523439    7037 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341\": container with ID starting with a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341 not found: ID does not exist" containerID="a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"
Oct 30 22:13:59.523556 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.523470    7037 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341"} err="failed to get container status \"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341\": rpc error: code = NotFound desc = could not find container \"a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341\": container with ID starting with a22fab5d3a134ac4f1371878e7ebf5cdf29a5a64e3b5f318676a1ed3a86c4341 not found: ID does not exist"
Oct 30 22:13:59.559272 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.559250    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:13:59.570205 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.570180    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8c3a04-57f4b89bb5-mzfjf"]
Oct 30 22:13:59.805734 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:13:59.805651    7037 scope.go:122] "RemoveContainer" containerID="a9bf2e57e0310f9df0a8b55cd1150c61b7633f43b1b6de7f1d7dc7d105a8567b"
Oct 30 22:14:00.705974 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:14:00.705942    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb37aea-8239-4308-8cba-65f378bc439e" path="/var/lib/kubelet/pods/4bb37aea-8239-4308-8cba-65f378bc439e/volumes"
Oct 30 22:14:03.671363 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:14:03.671326    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:14:33.671109 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:14:33.671067    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:14:59.846800 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:14:59.846723    7037 scope.go:122] "RemoveContainer" containerID="fb30f81746c179a8c0476a0944bef4c7a1dff6bdfce778133cd9fa70ff23ec3e"
Oct 30 22:15:02.877476 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:02.877441    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797c9987-ddv6q_ea5787a9-057e-48c5-9ca9-7c3f3400149c/authorino/0.log"
Oct 30 22:15:03.712660 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:15:03.712616    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:15:13.196323 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:13.196292    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9b9d764bc-9m9td_3837a8c5-b627-4a55-b2bc-7cfcede7214c/manager/0.log"
Oct 30 22:15:13.410376 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:13.410333    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-26nhx_cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b/manager/0.log"
Oct 30 22:15:14.627120 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:14.627087    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mmhps_6b818c11-e7eb-495d-b7f7-949553059316/manager/0.log"
Oct 30 22:15:15.793687 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:15.793653    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54fbf74c89-g26bb_b602ddb0-2bbd-460e-814a-53cb756561e5/maas-api/0.log"
Oct 30 22:15:16.386377 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:16.386346    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lp442_b81b9af7-2f76-48a3-9076-656ecd10cfb1/revoke-keys/0.log"
Oct 30 22:15:16.575985 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:16.575958    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-shgc9_401d6f8c-5480-410d-99a4-f81f4ffb2d8b/revoke-keys/0.log"
Oct 30 22:15:16.949385 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:16.949349    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3439857-dtmxq_a72cb22d-0df4-4d2d-9f6e-9b1228205d9a/revoke-keys/0.log"
Oct 30 22:15:18.999611 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:18.999573    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797c9987-ddv6q_ea5787a9-057e-48c5-9ca9-7c3f3400149c/authorino/0.log"
Oct 30 22:15:19.611337 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:19.611309    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-w87qg_249f3602-e4df-406a-ad76-f1e605158d4d/kuadrant-console-plugin/0.log"
Oct 30 22:15:21.654547 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:21.654518    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649f7bc66-trj7m_aab318c8-8afe-4e74-b82e-a85e5fd1e59a/kube-auth-proxy/0.log"
Oct 30 22:15:22.040569 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:22.040463    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bz96z_c9eea0c1-4200-4348-bdf4-5df5c5bab262/istio-proxy/0.log"
Oct 30 22:15:22.632350 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:22.632320    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-nkgpq_f538657b-f203-4b93-80d6-77522db77d56/istio-proxy/0.log"
Oct 30 22:15:23.633720 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:23.633688    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-xgns5_be548911-e631-4dac-85aa-591341df73f8/istio-proxy/0.log"
Oct 30 22:15:25.956328 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:25.956294    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nhzxz_c655b016-b634-484e-a598-4611ee2a6be6/main/0.log"
Oct 30 22:15:26.156898 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:26.156868    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2_5e30ce33-adeb-4935-91db-42bf140e78fe/main/0.log"
Oct 30 22:15:26.540922 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:26.540882    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb_2ba11244-cf05-4477-af93-89045c8c4bdd/main/0.log"
Oct 30 22:15:27.629195 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:27.629160    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797c9987-ddv6q_ea5787a9-057e-48c5-9ca9-7c3f3400149c/authorino/0.log"
Oct 30 22:15:29.031300 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:29.031267    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"]
Oct 30 22:15:29.041762 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:29.041730    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-shgc9"]
Oct 30 22:15:30.705932 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:30.705899    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401d6f8c-5480-410d-99a4-f81f4ffb2d8b" path="/var/lib/kubelet/pods/401d6f8c-5480-410d-99a4-f81f4ffb2d8b/volumes"
Oct 30 22:15:33.671268 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:15:33.671229    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:15:36.486270 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:36.486237    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9b9d764bc-9m9td_3837a8c5-b627-4a55-b2bc-7cfcede7214c/manager/0.log"
Oct 30 22:15:36.706637 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:36.706605    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-26nhx_cb6a26a4-4a55-41e5-9b59-1cc6fb32fc7b/manager/0.log"
Oct 30 22:15:37.756687 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:37.756658    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mmhps_6b818c11-e7eb-495d-b7f7-949553059316/manager/0.log"
Oct 30 22:15:38.697144 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:38.697112    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54fbf74c89-g26bb_b602ddb0-2bbd-460e-814a-53cb756561e5/maas-api/0.log"
Oct 30 22:15:39.084290 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:39.084259    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lp442_b81b9af7-2f76-48a3-9076-656ecd10cfb1/revoke-keys/0.log"
Oct 30 22:15:39.463248 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:39.463166    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3439857-dtmxq_a72cb22d-0df4-4d2d-9f6e-9b1228205d9a/revoke-keys/0.log"
Oct 30 22:15:40.965787 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:40.965755    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797c9987-ddv6q_ea5787a9-057e-48c5-9ca9-7c3f3400149c/authorino/0.log"
Oct 30 22:15:41.581630 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:41.581597    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-w87qg_249f3602-e4df-406a-ad76-f1e605158d4d/kuadrant-console-plugin/0.log"
Oct 30 22:15:43.370892 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:43.370859    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649f7bc66-trj7m_aab318c8-8afe-4e74-b82e-a85e5fd1e59a/kube-auth-proxy/0.log"
Oct 30 22:15:43.767213 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:43.767142    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bz96z_c9eea0c1-4200-4348-bdf4-5df5c5bab262/istio-proxy/0.log"
Oct 30 22:15:44.368059 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:44.368022    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-nkgpq_f538657b-f203-4b93-80d6-77522db77d56/istio-proxy/0.log"
Oct 30 22:15:45.378450 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:45.378410    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-xgns5_be548911-e631-4dac-85aa-591341df73f8/istio-proxy/0.log"
Oct 30 22:15:46.063732 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:46.063698    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"]
Oct 30 22:15:46.075103 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:46.075071    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lp442"]
Oct 30 22:15:46.706959 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:46.706923    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81b9af7-2f76-48a3-9076-656ecd10cfb1" path="/var/lib/kubelet/pods/b81b9af7-2f76-48a3-9076-656ecd10cfb1/volumes"
Oct 30 22:15:47.246779 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:47.246733    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nhzxz_c655b016-b634-484e-a598-4611ee2a6be6/main/0.log"
Oct 30 22:15:47.456346 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:47.456304    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh8m2_5e30ce33-adeb-4935-91db-42bf140e78fe/main/0.log"
Oct 30 22:15:47.849448 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:15:47.849400    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5m6mb_2ba11244-cf05-4477-af93-89045c8c4bdd/main/0.log"
Oct 30 22:16:03.671447 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:16:03.671400    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:16:09.026209 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:16:09.026124    7037 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"]
Oct 30 22:16:09.036718 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:16:09.036690    7037 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3439857-dtmxq"]
Oct 30 22:16:10.706520 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:16:10.706471    7037 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72cb22d-0df4-4d2d-9f6e-9b1228205d9a" path="/var/lib/kubelet/pods/a72cb22d-0df4-4d2d-9f6e-9b1228205d9a/volumes"
Oct 30 22:16:33.672451 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:16:33.672408    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:16:59.874948 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:16:59.874913    7037 scope.go:122] "RemoveContainer" containerID="fe54424f9e70461c59e4b833de62495473c4d2a0104bb477203a891676dbca83"
Oct 30 22:16:59.883514 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:16:59.883473    7037 scope.go:122] "RemoveContainer" containerID="587c7c66144fe2708ff5900e5e42707e1a975fdef18812f8a7d4199ad510e935"
Oct 30 22:17:03.671261 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:17:03.671221    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:17:10.435386 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:10.435353    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n5gz6_0bbf8b47-e869-43e4-8fdd-459639f068c3/global-pull-secret-syncer/0.log"
Oct 30 22:17:11.209365 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:11.209330    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9fkqb_7871fe29-fbbf-4733-878f-680fb56cee31/konnectivity-agent/0.log"
Oct 30 22:17:11.508246 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:11.508160    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-72.ec2.internal_ca4d5a6e1cac786a378b125c5389b555/haproxy/0.log"
Oct 30 22:17:18.299592 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:18.299558    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797c9987-ddv6q_ea5787a9-057e-48c5-9ca9-7c3f3400149c/authorino/0.log"
Oct 30 22:17:18.681180 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:18.681152    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-w87qg_249f3602-e4df-406a-ad76-f1e605158d4d/kuadrant-console-plugin/0.log"
Oct 30 22:17:20.265862 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.265808    7037 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"]
Oct 30 22:17:20.266287 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266239    7037 cpu_manager.go:414] "containerMap: removing container" podUID="4bb37aea-8239-4308-8cba-65f378bc439e" containerName="maas-api"
Oct 30 22:17:20.266287 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266250    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb37aea-8239-4308-8cba-65f378bc439e" containerName="maas-api"
Oct 30 22:17:20.266287 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266269    7037 cpu_manager.go:414] "containerMap: removing container" podUID="781cb114-f900-4962-97a3-9d185e6dad3e" containerName="payload-processing"
Oct 30 22:17:20.266287 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266274    7037 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781cb114-f900-4962-97a3-9d185e6dad3e" containerName="payload-processing"
Oct 30 22:17:20.266429 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266350    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="781cb114-f900-4962-97a3-9d185e6dad3e" containerName="payload-processing"
Oct 30 22:17:20.266429 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.266367    7037 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb37aea-8239-4308-8cba-65f378bc439e" containerName="maas-api"
Oct 30 22:17:20.270111 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.270089    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.290028 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.290001    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"]
Oct 30 22:17:20.296272 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.296215    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47t7f\"/\"kube-root-ca.crt\""
Oct 30 22:17:20.296447 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.296273    7037 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47t7f\"/\"openshift-service-ca.crt\""
Oct 30 22:17:20.339958 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.339923    7037 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47t7f\"/\"default-dockercfg-5hv7n\""
Oct 30 22:17:20.356428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.356393    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-proc\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.356428 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.356433    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-podres\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.356629 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.356619    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcj9\" (UniqueName: \"kubernetes.io/projected/62e3b9f6-7f84-4ee9-b860-47b680810fbe-kube-api-access-6wcj9\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.356669 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.356649    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-sys\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.356758 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.356735    7037 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-lib-modules\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458090 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458050    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcj9\" (UniqueName: \"kubernetes.io/projected/62e3b9f6-7f84-4ee9-b860-47b680810fbe-kube-api-access-6wcj9\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458090 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458098    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-sys\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458141    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-lib-modules\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458165    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-proc\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458354 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458185    7037 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-podres\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458869 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458851    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-podres\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458869 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458864    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-lib-modules\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458999 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458851    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-proc\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.458999 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.458850    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62e3b9f6-7f84-4ee9-b860-47b680810fbe-sys\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.479634 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.479604    7037 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcj9\" (UniqueName: \"kubernetes.io/projected/62e3b9f6-7f84-4ee9-b860-47b680810fbe-kube-api-access-6wcj9\") pod \"perf-node-gather-daemonset-mpg9w\" (UID: \"62e3b9f6-7f84-4ee9-b860-47b680810fbe\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.531251 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.531176    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/alertmanager/0.log"
Oct 30 22:17:20.583359 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.583314    7037 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:20.666473 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.665817    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/config-reloader/0.log"
Oct 30 22:17:20.808279 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.808198    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/kube-rbac-proxy-web/0.log"
Oct 30 22:17:20.879647 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.879621    7037 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"]
Oct 30 22:17:20.881478 ip-10-0-137-72 kubenswrapper[7037]: W1030 22:17:20.881445    7037 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod62e3b9f6_7f84_4ee9_b860_47b680810fbe.slice/crio-ddba12e748f6198b36e3c0d6dd03942a8c59dd0631f696745c27530a2c08a689 WatchSource:0}: Error finding container ddba12e748f6198b36e3c0d6dd03942a8c59dd0631f696745c27530a2c08a689: Status 404 returned error can't find the container with id ddba12e748f6198b36e3c0d6dd03942a8c59dd0631f696745c27530a2c08a689
Oct 30 22:17:20.935430 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:20.935406    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/kube-rbac-proxy/0.log"
Oct 30 22:17:21.059644 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.059545    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/kube-rbac-proxy-metric/0.log"
Oct 30 22:17:21.199344 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.199306    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/prom-label-proxy/0.log"
Oct 30 22:17:21.321936 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.321913    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db02ac6c-2f97-4944-b217-e48574269a40/init-config-reloader/0.log"
Oct 30 22:17:21.594861 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.594780    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5n2m_548cadb6-b5c0-4f7d-ad73-e5060de08fc9/kube-state-metrics/0.log"
Oct 30 22:17:21.618404 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.618369    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w" event={"ID":"62e3b9f6-7f84-4ee9-b860-47b680810fbe","Type":"ContainerStarted","Data":"38355852c23f187ba4cd4e5c81888e888a4cb6f6d76859ed8ca74ff669219d82"}
Oct 30 22:17:21.618624 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.618429    7037 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w" event={"ID":"62e3b9f6-7f84-4ee9-b860-47b680810fbe","Type":"ContainerStarted","Data":"ddba12e748f6198b36e3c0d6dd03942a8c59dd0631f696745c27530a2c08a689"}
Oct 30 22:17:21.618624 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.618456    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:21.714321 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.714278    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5n2m_548cadb6-b5c0-4f7d-ad73-e5060de08fc9/kube-rbac-proxy-main/0.log"
Oct 30 22:17:21.838288 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.838255    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5n2m_548cadb6-b5c0-4f7d-ad73-e5060de08fc9/kube-rbac-proxy-self/0.log"
Oct 30 22:17:21.971886 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:21.971795    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7dc8b9966c-xmpxj_ce43bf79-f468-4826-ae78-e58f103f9fa7/metrics-server/0.log"
Oct 30 22:17:22.096578 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:22.096547    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-cd2qc_8211ee52-aa1e-4a0c-bfe8-3e1779da22a7/monitoring-plugin/0.log"
Oct 30 22:17:22.627022 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:22.626993    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jdnjm_6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/node-exporter/0.log"
Oct 30 22:17:22.749189 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:22.749154    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jdnjm_6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/kube-rbac-proxy/0.log"
Oct 30 22:17:22.871502 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:22.871468    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jdnjm_6c61d82e-9c9a-49b4-a6b1-fa6b44a97e19/init-textfile/0.log"
Oct 30 22:17:23.369448 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.369415    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p5f87_e8062ab8-4d4e-41a1-9ed0-1328ffc23870/kube-rbac-proxy-main/0.log"
Oct 30 22:17:23.491193 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.491163    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p5f87_e8062ab8-4d4e-41a1-9ed0-1328ffc23870/kube-rbac-proxy-self/0.log"
Oct 30 22:17:23.614391 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.614359    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p5f87_e8062ab8-4d4e-41a1-9ed0-1328ffc23870/openshift-state-metrics/0.log"
Oct 30 22:17:23.746565 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.746470    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/prometheus/0.log"
Oct 30 22:17:23.867719 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.867693    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/config-reloader/0.log"
Oct 30 22:17:23.987384 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:23.987352    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/thanos-sidecar/0.log"
Oct 30 22:17:24.110256 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.110221    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/kube-rbac-proxy-web/0.log"
Oct 30 22:17:24.234912 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.234882    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/kube-rbac-proxy/0.log"
Oct 30 22:17:24.357593 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.357561    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/kube-rbac-proxy-thanos/0.log"
Oct 30 22:17:24.483339 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.483268    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_78e92c1a-1cea-405a-9691-6c490e6c8d32/init-config-reloader/0.log"
Oct 30 22:17:24.632023 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.631995    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qmv5z_87e040ec-3520-40f4-b906-cc22de4d532a/prometheus-operator/0.log"
Oct 30 22:17:24.769083 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:24.768995    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qmv5z_87e040ec-3520-40f4-b906-cc22de4d532a/kube-rbac-proxy/0.log"
Oct 30 22:17:25.425816 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:25.425783    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/thanos-query/0.log"
Oct 30 22:17:25.546282 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:25.546237    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/kube-rbac-proxy-web/0.log"
Oct 30 22:17:25.668344 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:25.668299    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/kube-rbac-proxy/0.log"
Oct 30 22:17:25.790081 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:25.789997    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/prom-label-proxy/0.log"
Oct 30 22:17:25.911717 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:25.911686    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/kube-rbac-proxy-rules/0.log"
Oct 30 22:17:26.032992 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:26.032948    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fd47999b4-r2bv9_943e6093-44ff-4e3e-97d4-67a24ce560a0/kube-rbac-proxy-metrics/0.log"
Oct 30 22:17:27.634589 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:27.634561    7037 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w"
Oct 30 22:17:27.670897 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:27.670843    7037 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-mpg9w" podStartSLOduration=7.670830932 podStartE2EDuration="7.670830932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:21.661453264 +0000 UTC m=+4585.072992701" watchObservedRunningTime="2026-10-30 22:17:27.670830932 +0000 UTC m=+4591.082370363"
Oct 30 22:17:33.680471 ip-10-0-137-72 kubenswrapper[7037]: E1030 22:17:33.680423    7037 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zztpp" containerName="extractor"
Oct 30 22:17:33.765526 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:33.765475    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lr5s_1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/dns/0.log"
Oct 30 22:17:33.820528 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:33.820473    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5649f7bc66-trj7m_aab318c8-8afe-4e74-b82e-a85e5fd1e59a/kube-auth-proxy/0.log"
Oct 30 22:17:33.885303 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:33.885269    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lr5s_1e46a0b9-72ff-4aab-8f50-51dc54ee0b0c/kube-rbac-proxy/0.log"
Oct 30 22:17:33.950112 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:33.950016    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bz96z_c9eea0c1-4200-4348-bdf4-5df5c5bab262/istio-proxy/0.log"
Oct 30 22:17:34.153602 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:34.153561    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-nkgpq_f538657b-f203-4b93-80d6-77522db77d56/istio-proxy/0.log"
Oct 30 22:17:34.466099 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:34.466062    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-xgns5_be548911-e631-4dac-85aa-591341df73f8/istio-proxy/0.log"
Oct 30 22:17:34.525481 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:34.525455    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rh2wd_2b36017a-ec01-4979-a358-7198261684d7/dns-node-resolver/0.log"
Oct 30 22:17:36.030710 ip-10-0-137-72 kubenswrapper[7037]: I1030 22:17:36.030638    7037 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfthv_c56952e8-b525-401d-8128-e4c8ac0d6469/node-ca/0.log"